MEEP configuration specific to Courtney

MEEP: MIT Electromagnetic Equation Propagation
Written by Steven Johnson, those of you already familiar with MPB can begin using this pretty quickly. Their webpage gives many examples and tutorials but I will give a few common features here and outline usage for MEEP's MPI capabilities on courtney.

To invoke meep, create a ctl file and invoke from your n1ge6 submit script: 

meep input.ctl
Microdisk CTL file
We let the standard length unit "a" be 1um. Thus: 
(define-param n 3.3) ; index of waveguide
(define-param r 1)   ; radius of disk (1 um)
(define-param pad .81)   ; padding between disk and edge of PML
(define-param dpml .144) ; thickness of PML (16 layers)
(set-param! resolution 111) ; 111 pixels per length unit => dx=9nm

(define ie (* 2 (+ r pad dpml))) ; total grid size
(set! geometry-lattice (make lattice (size ie ie no-size)))
(set! geometry (list
                (make cylinder (center 0 0) (height infinity)
                      (radius r) (material (make dielectric (index n))))))
(set! pml-layers (list (make pml (thickness dpml))))

(define-param lambda0 .896) ; pulse center wavelength
(define-param df .025)  ; pulse width (in frequency)
(set! sources (list
               (make source
                 (src (make gaussian-src (wavelength lambda0) (fwidth df)))
                 (component Ez) (center 0 0))
               ))
; by default the Courant number is S=0.5 and c=1
; making dt=S*dx/c=0.0045
; so 310,000 time steps is approximately 1400 s in these units:
(run-until 1400
          (at-beginning output-epsilon)
          (at-end output-efield-z))
We can also tell MEEP to run a certain amount of time after the source has disappeared, thus making harminv's analysis more reliable. To do this we can replace the run-until line with run-sources+ 
(run-sources+ 1000
             (after-sources (harminv Ez (vector3 0) (/ 1 lambda0) df)))
which will search for modes centered around our excitation wavelength with a frequency range of df.

Another powerful feature of meep is its ability to automatically take advantage of the symmetry of your system. It was not done above, but adding the command: 

(set! symmetries (list (make mirror-sym (direction Y))))
will exploit a mirror-symmetry plane through the origin perpendicular to the y direction. This symmetry forcing operate on both your sources and material. If your system does not actually have this symmetry, you will not get the results you desire.
There are other examples located in /usr/local/doc/meep/ on janelle.

Using MPI with MEEP
To invoke meep-mpi from a submit script for n1ge6, simply use: 

mpiexec -machinefile $TMPDIR/machines -n $NSLOTS meep-mpi input.ctl
(with all the other required options for n1ge6 included as well). You should not have to modify your ctl file whatsoever in order to run meep-mpi. There are a few things to consider, however:
  • The HDF5 libraries installed on courtney do not support parallel I/O. Thus, if you are writing the electric field (or any other function utilizing the HDF5 libraries) in frequent intervals, the computation will slow down quite a bit. There are, apparently, hacks in place to write HDF5 files using serial I/O for meep-mpi but its stability is not guaranteed so it is probably in your best interests to use the serial version of meep for these cases.
  • Because of the limited bandwidth of Courtney's LAN, runtime does not scale linearly with the number of processors you use for meep-mpi. Secondly, any FDTD grid will not be homogeneous if you include a PML, usually making cell division non-uniform. And lastly, because of the lack of homogeneity of courtney, different machines chosen by the scheduler will result in a slightly different runtime. Below is a graph showing some averaged results of running the microdisk file above with meep-mpi.

    It seems that 4 processors is an optimal choice for utilizing the cluster's resources most efficiently. This is, of course, highly dependent on your particular configuration. For larger grid sizes, processing time is likely to increase more than communication time allowing for the efficient use of more than 4 processors.

    If you are curious why mpb-mpi is not installed, the FFTW libraries mpb uses are very communication intensive requiring a faster network (like gigabit) than Courtney has.

    Data Processing
    shot,show and snap have been modified slightly to include .h5 files. So, any h5 output from meep can be converted to png using them. To see what is happening verbosely use the flag -v. To use another colormap type -c colormap (default is bluered). To use another overlay file with shot (default is structure.dat) invoke -C overlayfile. If you are trying to convert .h5 files, the default overlay file is any file located in your current working directory containing "eps" in the name. 

    Example: shot -v disk-ez-*.h5 -c dkbluered
    This will overlay the contour of disk-eps.h5 onto all of the output field profiles. You can add an alias to your login file (.bashrc or .cshrc) to always use the colormap of your choice: 
    alias shot "shot -c vga"