XMM-Newton & Chandra: analysis scripts @ Lecce

Introduction to scripting

The following C-shell scripts were written in order to reduce and analyse XMM-Newton data in a fast and simple way. I recommend the reader to check what one particular script does, understand it and possibly adapt it for his/her needs. Possibly, bugs are out there, so be careful!

The scripts make use of SAS tasks and HEASOFT's FTOOLS utilities. Many of them combine several SAS commands together and call other C-shell scripts in order to do a given task. Even if we tried to write the scripts as general as possible, there are default values that may not fit your standard. Moreover, the codes are always under development - let's say - the codes on this web-page are not necessarily the latest and goodest ones. So, do not hesitate to modify the codes: improvements are welcome.

As usual, before starting a given script, the SAS (and/or CIAO) environment has to be started up correctly. We address the reader to the SAS and CIAO web-pages where further details can be found.

A last note before starting: it is always a good idea to redirect the output of a script to a log file, using tee. i.e.:
script.csh argument1 argumument2 ... argumentN | & tee script_log.txt
In this case, you will have memory of what the script did as well as the input arguments you have used.

To download an individual script, simply click on it. A zipped file containing all the scripts is also available here.

Note also that a few codes are written in IDL and Fortran. Just to make things easier!

Return to top

Description of Individual Scripts

• count_files.csh - version 1.0
  • Example script: Count the files with a given extension.
    
    The script is invoked as:
    source ./count_files.csh txt
    


• list_things.csh - version 1.0
  • Example script: List folders, files (or everything) which is contained in the calling directory. The list is done with full path and the output written onto an ASCII file.
    
    The script is invoked as:
    source ./list_things.csh [F,f,A]
    where F, f and A stand for Folders, Files and ALL.
    


• sync_folders.csh - version 1.0
  • Example script: Since I use to work indifferently on several Linux machines, I have the necessity to synchronize folders and files. The same can be done (automatically) with AFS, but if AFS is not there this simple script (based on rsync call) is very useful.
    
    The script is invoked as:
    source ./sync_folders.csh LocalFolder RemoteFolder RemoteUser RemoteHost Flag"
    Flag is 0 or 1 depending if you want to copy from Local to Remote or viceversa
    


• remove_entries.csh - version 1.0
  • Given an ASCII list with a column of sequential numerical labels (i.e. 0,1,2...N) and a list of numerical flags, the script removes all the entries in the list associated to that flags. The input keywords DS9 yes (or no) selects if the input list is a suitable DS9 region file and acts consequently. Useful to purge ASCII tables with undesired entries.
    
    The script is invoked as:
    source ./remove_entries.csh FlagList InputList OutputList DS9 yes (no)
    


• make_regions.pro - version 2.0
  • Given a FITS list of sources (as identified in MOS1, MOS2 and pn) with coordinates and ID source numbers (as determined by the edetect_chain SAS tool for each camera separately) write a DS9 region file with color codes. -1 in the camera ID source number column means that the source was not identified in that camera. The code expects that the FITS file contains the following columns
    CAT_X_RA ; Ra (deg)
    CAT_X_DEC ; Dec (deg)
    MOS1 ; MOS 1 index
    MOS2 ; MOS 2 index
    PN ; pn index
    FALL ; Flux
    
    
    The script is invoked as:
    IDL> make_regions, InputFile=InputFile,Fmin=Fmin
    where Fmin is the minimum flux to be considered, i.e. only sources in the catalogue with Flux ge Fmin will be inserted in the final DS9 region file.


• make_ODF_folder.csh - version 1.0
  • Once the data file (*.TAR) has been downloaded locally by the XSA tool, the scripts extracts the useful data files and writes an input.txt file suitable for the subsequent .csh scripts (particularly sasin.csh)
    
    The script is invoked as:
    source ./make_ODF_folder.csh TARFILE
    


• sasini.csh - version 1.0
  • Initialize the SAS environment so that the analysis on a given XMM-Newton data set can be started. Once SAS is on, the scripts runs ccfbuild and odfingest (if not already done). The script uses a .txt file as input formatted as:
    
    OBSID_0305970101
    0305970101
    100
    100
    100
    100
    path to odf folder
    
    where the entries are: dummy entry (line #1), obs id (line #2), dummy entries (line #3, #4, #5), path to the ODF folder (line #6).
    
    The script is invoked as:
    source ./sasini.csh input.txt local
    
    Note that the scripts requires the set of the variables 1) ${SAS_INSTALLATION_FOLDER} (pointing at the folder where setsas.csh is located); 2) ${SAS_CCFPATH} (pointing at the folder containing the calibration files); You can set the variables in a suitable log-in file as .cshrc.
    
    
• make_aper.csh - version 1.0 (This script is not mantained)
  • Make the aperture photometry of a source (or list of sources) by using the funtools codes (already installed). Note that the scripts requires the list.txt file to be formatted as:
    
    Xsource|Ysource|RADsource|XBck|YBck|RadBck int|RadBck ext
    
    where in each line one has the coordinates of the source and backgrund extraction radius. The background is extracted in an annulus around the source.
    
    The script is invoked as:
    source ./make_aper.csh image.fits list.txt
    
    
• make_radial.csh - version 1.0 (This script is not mantained)
  • Make the radial profile a source by using the funtools codes (already installed).
    
    The script is invoked as:
    source ./make_radial.csh image.fits Xsource Ysource RadMin Radmax N Radbck
    
    where Xsource, Ysource, RadMin, Radmax, N, and Radbck are the coordinates of the source, the minimum and maximum extraction radius for the source, the number of annuli to be used, the extraction radius of the background (Radbck > Radmax).
    
    
• make_aperture_photometry - version 2.0
  • Make the aperture photometry of a source by using SAS tasks for a given source. It extracts the source and background light curves and calculate the signal to noise ratios for a given set of extraction radii. Useful if one wants to maximize the extraction radius of a given source w.r.t. that of the background.
    
    The code is a parserer to the scripts make_aperture_ph (in csh) and make_aperture_ph.pro (in IDL). Please, read teh header of each code for further details on the input parameters and on the way the code works.
    
    The script is invoked as:
    source ./make_aperture_photometry.csh
    
    
• extract_nh.csh - version 1.0
  • Extract the NH value in a given direction of the sky. Uses HEASOFT.
    
    The script is invoked as:
    source ./extract_nh.csh RA DEC
    
    
• make_catalogue.csh - version 1.0
  • Extract the coordinates the sources found in the USNO-B1 catalogue around targed and in a given radius. The script runs an IDL code (make_catalogue.pro ) and the internet connection has to be ON since the VIZIER site is queryed. In output is a .fits file (suitable for the SAS task eposcorr).
    
    The script is invoked as:
    source ./make_catalogue.csh ${work} "'USNO-B1'" "'cat_USNOB1.fits'" RA DEC DIST
    
    where RA and DEC are in sexagesimal degrees and DIST is in arcminutes. Catalogues different than USNO-B1 are in principle good for the code. However, the user is adviced that the VIZIER query syntax depends on the selected catalogue. Hence, minor changes are sometimes required in the IDL script.



• make_source_detection.csh - version 1.0
  • Perform the source detection in one of the EPIC camera in the standard energy bands used in The XMM-Newton Serendipitous Source Catalogue: 2XMMi-DR3. The energy conversion factors are updated to April 28, 2010. The ECFs enter into the script via the (ecf.txt ) file which contains the updated ECFs for the MOS1, MOS2 and pn (in different filters - THIN, MEDIUM and THICK). The ECFs have been already multiplied by corrective factors in order to account for the latest calibrations (see here).
    
    The script is invoked as (e.g.):
    source ./make_source_detection.csh MOS1 MEDIUM mos1_clean_event_file output_root output_folder
    
    
• slave_script_source_detection.csh - version 3.0
  • A (more or less) complete script for source detection. It calls sasin.csh, make_source_detection.csh, make_catalogue.csh and performs the SAS task eposcorr to correct the source coordinates.
    
    The script is invoked as:
    source ./slave_script_source_detection.csh
    
    
• check_pile_up.csh - version 1.0 (Experimental)
  • A script to test the pile up of a given source (based on the SAS threads). It first extracts an event file for the given source (applying gtis) and recursively cut the central region of the PSF until the statistics of singles equals the statistics of doubles. The script runs an IDL code (check_pile_up.pro ) for plot purposes.
    
    The script is invoked as:
    source ./check_pile_up.csh event_list_file gti_file output_root X Y Rad MinRad MaxRad N
    
    where event_list_file, gti_file, output_root, X,Y, RAD, Minrad, MaxRad, and N are the input (RAW) event file, the good time interval file (gti) which flags the high energy background intervals, the root of the output names, the source X and Y coordinates, the radius of the region around the source, the minimum and maximum radii of the central circular part of the PSF to exclude, the number of regions to test. Note that the coordinates are in image system. The script also returns the file epatplot.txt containing the statistics for singles, error on singles, statistics on doubles, error on doubles and extraction radius.


• make_clean.csh - version 2.0
  • A script to clean an XMM-newton data set from high background activity periods and to produce adequate event cleaned files. For checking purposes, several diagnostic fits files are produced and useful information are written on screen. The script can be used in MANUAL or AUTO modes. The AUTO mode uses the script (xmmautoclean.csh ) which derives from the script xmmlight_clean.csh (from the Birmingham XMM scripts) to which we refer for more details. Essentially, the AUTO mode purges an high energy light curve until a 3 sigma criterion is satisfied.
    
    The script is invoked as:
    source ./make_clean.csh MOS[PN] InputEventList MANUAL[AUTO] BIN OutpuRoot RATE[COUNT] EMIN EMAX
    
    where InputEventList, MANUAL[AUTO], BIN, OutputRoot, RATE[COUNT], EMIN, and EMAX are the input (RAW) event file, the cleaning mode, the bin size for the histograms, the output file root, the rate-count mode for the checking histograms, and the minimum and maximum energies to be used for the high energy light curves. The script writes useful information on screen. The output -cleaned- files (including the diagnostic fits) are written in the same folder.


• make_image.csh - version 2.0
  • A script to create an image in a given band and with given filter expression. If RGB is set, three images in different standard bands (see the code to fix the bands) are produced and then displayed on screen via Ds9. Note that the input event list has to be already cleaned.
    
    The script is invoked as:
    source ./make_image.csh MOS[PN] InputEventList Emin Emax Bin pattern OutputRoot RGB
    
    where InputEventList, MOS[PN], BIN, OutputRoot, EMIN, EMAX and RGB are the input (possibly cleaned) event file, the EPIC camera (for filter purposes), the bin size, the output file root, and the minimum and maximum energies to be used for the image.


• make_BaryCenCorrectionLC.csh - version 1.0
  • The script performs the barycenter correction to a list of light curves as produced by the SAS tasks.
    
    The script is invoked as:
    source ./make_BaryCenCorrectionLC.csh list.txt
    
    where list.txt contains the name of the files associated to the light curves to correct. If the light curve has been produced by teh SAS tasks the fits file contain the tabel RATE whose column TIME is modified in output!


• make_fakeit.csh - version 1.0
  • The script simulates a spectrum via the XSPEC package and calculate the counts (or fluxes) in selected bands (useful for HR evaluation). Follow the instructions inside to adapt the script to you case (the code uses a PHABS*(POWER) within XSPEC). The input spectrum can be grouped (with useful BACKFILE, RESPFILE and ANCRFILE keyword already set) or not. In the latter case, the code produces a spectrum.fits file with the right keyword and with data grouped by 25 count/bin. Then faked spectrum is produced.
    
    The script is invoked as:
    source ./make_fakeit.csh Spectrum Background Response Ancillary Output
    
    


• make_fakeit_loop.csh - version 1.0
  • The script simulates several spectra via the XSPEC package and calculate the counts (or fluxes) in selected bands (useful for HR evaluation). Follow the instructions inside to adapt the script to you case (the code uses a PHABS*(POWER) and PHABS*(BREMSS) within XSPEC). The input spectrum can be grouped (with useful BACKFILE, RESPFILE and ANCRFILE keyword already set) or not. In the latter case, the code produces a spectrum.fits file with the right keyword and with data grouped by 25 count/bin. Then faked spectra are produced. Ranges for the hydrogen column density and power law index are entered on run-time.
    
    The script is invoked as:
    source ./make_fakeit_loop.csh Spectrum Background Response Ancillary Output
    
    


• make_fakeit_loop.pro - version 1.0
  • IDL code to produce nice plots of observed HRs data overplotted on expected HRs as resulted from simulations, via e.g. make_fakeit_loop.csh.
    
    The script is invoked as:
    IDL> status=make_fakeit_loop(infile=infile,outfile=outfile, hrdat=hrdat)
    where infile, outfile, and hrdat are the input file for the simulated HRs (as produced by make_fakeit_loop.csh), the output file name of the .ps file and the input observed HRs as produced by make_combine_list.pro, respectively.
    


• make_xmm_lc.pro - version 2.0
  • This code (still experimental) is an IDL program which constructs a background subtracted light curve for a given source. The code requires that you have already saved the source and background events in suitable fits files by using SAS. It also requires the BACKSCAL values for the source and background extraction regions as found in suitable spectra files. Essentially, the code finds the absolute minimum and maximum times of the events (in common to all the cameras) and construct a light curve via a histogram of the arrival times. We address the reader to the comments inserted in the header of the code for further details.
    
    The script is invoked as:
    IDL> status=make_xmm_lc(filename=filename, filepath=filepath, binsize=binsize, RANGEPI=RANGEPI,RANGEPATTERN=RANGEPATTERN,$ mode=mode, time=time, rate=rate, error=error,plot=plot)
    
    


• read_xmm_events.pro - version 3.0
  • This code is an IDL program which reads the events file for the source (and or background) as produced by SAS. It the extracts useful informations as the event lists, PI, PATTERN, GTIs, T_exp, starting and ending arrival times. We address the reader to the comments inserted in the header of the code for further details.
    
    The script is invoked as:
    IDL> status= read_xmm_events(FILENAME=FILENAME, FILEPATH=FILEPATH,RANGEPI=RANGEPI,RANGEPATTERN=RANGEPATTERN, $ T0=T0, TMIN=TMIN, TMAX=TMAX, TSTART=TSTART, TSTOP=TSTOP, T_EXP=T_EXP, STDGTI=STDGTI, $ TIME=TIME, PI=PI,PATTERN=PATTERN)
    
    


• read_xmm_epiclccorr.pro - version 3.0
  • This code is an IDL program which reads the light curves (background corrected, epiclccorr adjusted and eventually barycenter corrected) as produced by SAS and load the results in suitable arrays for further analysis. Note that the light curves of the single cameras have to be already syncrhronized, i.e. produced between the same minimum and maximum time and with the same bin size. This can be easily done in SAS. We address the reader to the comments inserted in the header of the code for further details.
    
    The script is invoked as:
    IDL> status=read_xmm_epiclccorr("inputfile", fileroot=fileroot, time=time,rate=rate,errate=errate, tmin=tmin, tmax=tmax, mode=mode)
    
    


• make_interpol.for - version 2.0
  • A (sometimes) useful code to interpol X,Y(X) data on a new grid of X' points. Please, read the header for the input parameters.
    
    The script is invoked as:
    IDL> g77 make_interpol.for -o make_interpol.exe
    
    


• bbfit.pro - version 1.0
  • This package contains all the routines required to make an easy Black-Body or Grey-Body fit to the data collected in the infrared band (Hershel-like bands...see the benchmark within the code).
    The package contains a set of useful functions. In particular,
    1) mybb(X, P) useful for the construction of a black body function. The function is build so that it avoids "large numbers" and depends only on two main parameters:
    P[0] = Normalization in units of 10^-10 P[1] = The temperature in kelvin
    The input data (X) have to enter in microns which implies that the output model is in Jansky. Thus the package can be used to fit data entering in the following units: (micron, jansky)
    2) mygb(X, P) useful for the construction of a grey body function. The function is build so that it avoids "large numbers" and depends only on two main parameters:
    P[0] = Normalization in units of 10^-10 P[1] = The temperature in kelvin P[2] = beta factor
    The input data (X) have to enter in microns which implies that the output model is in Jansky. Thus the package can be used to fit data entering in the following units: (micron, jansky)
    
    The script is invoked as, e.g.:
    IDL> .compile bbfit
    IDL> wavelenght= [160D0,250D0,350D0,500D0] ; Microns, Note that the input have to be in double!
    IDL> fluxes=[3.61369D0,2.91990D0,1.47529D0,0.601934D0] ; Jansky, Note that the input have to be in double!
    IDL> WEIGTHS=[1,1,1,1]
    IDL> make_bb_fit, wavelenght=wavelenght, fluxes=fluxes, WEIGHTS=WEIGHTS,Nbb=Nbb,tbb=tbb, /quiet ; Provide a Black body fit. No print in output
    IDL> make_gb_fit, wavelenght=wavelenght, fluxes=fluxes, WEIGHTS=WEIGHTS,Ngb=Ngb,tgb=tgb, beta=beta,/quiet ; Provide a Grey body fit. No print in output
    
    Notes:
    starting points of the fit minimization are automatically selected, unless the keywords
    startbb=[Nbbstart, Tbbstart] for the Black Body fit
    startgb=[Ngbstart, Tgbstart, Betastart] for the Grey Body fit
    are given.
    
    

Return to top

---

A.A. Nucita
This page contains analysis scripts which are provided "as they are". They do not represent the goodest solution to a given task, but may be consider as a good starting point.
The Team (and inputs from all our co-workers)
Last modified: June 08 2012