Can't print entire data set with fits from astropy.io - python-3.x

Question

I have a large fits file (21.4 MB). I would like to print the contents of it to a text file, but can only access a portion of it. I am looking for help getting the entire file to text format.
> from astropy.io import fits
> hdulist = fits.open('N20190326G0041i.fits')
Information on the file. Note that everything is in the primary HDU.
> hdulist.info()
Filename: N20190326G0041i.fits
No. Name Ver Type Cards Dimensions Format
0 PRIMARY 1 PrimaryHDU 183 (190685, 28) float32
I can access the full header but it is extremely long. I included it at the end of this post.
> hdu = hdulist[0]
> hdu.header
However, I only get a portion of the data using hdu.data
> hdu.data
array([[ 4.0630740e+02, 4.0631021e+02, 4.0631290e+02, ...,
1.0478779e+03, 1.0478831e+03, 1.0478882e+03],
[ 2.7955999e+01, 3.1493999e+01, 1.2378000e+01, ...,
-4.3614998e+00, -1.8785000e+00, -8.8672000e-01],
[ 2.8534999e+00, 2.8862000e+00, 2.9282999e+00, ...,
-6.1020999e+00, -5.2989998e+00, -5.1680999e+00],
...,
[ 1.7951000e+04, 2.9099000e+04, 3.5257000e+03, ...,
1.0594000e+03, 7.9347998e+02, 1.6349001e+02],
[ 3.1568999e+03, 3.1631001e+03, 3.2426001e+03, ...,
3.2828000e+02, 3.2062000e+02, 3.2189001e+02],
[ 3.3338000e+03, 3.3806001e+03, 3.4557000e+03, ...,
2.1803000e+02, 2.2574001e+02, 2.3003999e+02]], dtype=float32)
What I typically do to print fits files to text files is ...
> table = hdulist[0].data
> print(table, file = open('test.txt','a'))
This "works", and outputs the same excerpt of the data that hdu.data prints on screen.
> [[ 4.0630740e+02 4.0631021e+02 4.0631290e+02 ... 1.0478779e+03
1.0478831e+03 1.0478882e+03]
[ 2.7955999e+01 3.1493999e+01 1.2378000e+01 ... -4.3614998e+00
-1.8785000e+00 -8.8672000e-01]
[ 2.8534999e+00 2.8862000e+00 2.9282999e+00 ... -6.1020999e+00
-5.2989998e+00 -5.1680999e+00]
...
[ 1.7951000e+04 2.9099000e+04 3.5257000e+03 ... 1.0594000e+03
7.9347998e+02 1.6349001e+02]
[ 3.1568999e+03 3.1631001e+03 3.2426001e+03 ... 3.2828000e+02
3.2062000e+02 3.2189001e+02]
[ 3.3338000e+03 3.3806001e+03 3.4557000e+03 ... 2.1803000e+02
2.2574001e+02 2.3003999e+02]]
Also, I repeated all of the above things using memmap = True, but get the same results.
> from astropy.io import fits
> hdulist = fits.open('N20190326G0041i.fits', memmap = True)
I also tried the convenience functions, but that produced the exact same excerpt as hdu.data.
> tbdata = fits.getdata('N20190326G0041i.fits')
> print(tbdata,file=open('test.txt','a'))
I also tried the astropy.table package, but could not get it to work either.
> from astropy.table import Table
> t = Table.read(hdulist[0], format = 'fits')
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/anaconda3/lib/python3.7/site-packages/astropy/table/connect.py", line 52, in __call__
out = registry.read(cls, *args, **kwargs)
File "/anaconda3/lib/python3.7/site-packages/astropy/io/registry.py", line 523, in read
data = reader(*args, **kwargs)
File "/anaconda3/lib/python3.7/site-packages/astropy/io/fits/connect.py", line 195, in read_table_fits
memmap=memmap)
File "/anaconda3/lib/python3.7/site-packages/astropy/io/fits/hdu/hdulist.py", line 151, in fitsopen
lazy_load_hdus, **kwargs)
File "/anaconda3/lib/python3.7/site-packages/astropy/io/fits/hdu/hdulist.py", line 390, in fromfile
lazy_load_hdus=lazy_load_hdus, **kwargs)
File "/anaconda3/lib/python3.7/site-packages/astropy/io/fits/hdu/hdulist.py", line 1039, in _readfrom
fileobj = _File(fileobj, mode=mode, memmap=memmap, cache=cache)
File "/anaconda3/lib/python3.7/site-packages/astropy/utils/decorators.py", line 521, in wrapper
return function(*args, **kwargs)
File "/anaconda3/lib/python3.7/site-packages/astropy/io/fits/file.py", line 180, in __init__
self._open_filelike(fileobj, mode, overwrite)
File "/anaconda3/lib/python3.7/site-packages/astropy/io/fits/file.py", line 533, in _open_filelike
"method, required for mode '{}'.".format(self.mode))
OSError: File-like object does not have a 'write' method, required for mode 'ostream'.
However, if I use
> t=Table.read(hdu.data, format='fits')
then I get a different error
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/anaconda3/lib/python3.7/site-packages/astropy/table/connect.py", line 52, in __call__
out = registry.read(cls, *args, **kwargs)
File "/anaconda3/lib/python3.7/site-packages/astropy/io/registry.py", line 523, in read
data = reader(*args, **kwargs)
File "/anaconda3/lib/python3.7/site-packages/astropy/io/fits/connect.py", line 195, in read_table_fits
memmap=memmap)
File "/anaconda3/lib/python3.7/site-packages/astropy/io/fits/hdu/hdulist.py", line 147, in fitsopen
if not name:
ValueError: The truth value of an array with more than one element is ambiguous. Use a.any() or a.all()
I have also tried tprint and tdump in PyRAF, it simply gives the error "Warning: ", with no other helpful information. I also tried writing the fits file to text using the wspectext function in PyRaf, but it is unable to do so.
However, I want to be able to just use the fits package from astropy to output the data to a text file. I have done this countless times with other fits files from a variety of telescope pipelines, but it isn't working this time. Any help is much appreciated.
The header of the file is below. I'm wondering since the header was written in IDL if the only way I'll be able to access the info is through IDL? This seems unlikely to me. I'd really rather avoid using IDL if possible. We have a site license at our university, but we are not permitted on campus during the pandemic and our VPN capabilities are lacking.
> hdu.header
SIMPLE = T / Written by IDL:
BITPIX = -32 / Real*4 (complex, stored as float)
NAXIS = 2 / Number of axes
NAXIS1 = 190685 / Number of pixel columns
NAXIS2 = 28 / Number of pixel rows
INHERIT = F / No need to inherit global keywords
BZERO = 0. / Data is Unsigned Integer
BSCALE = 1. / Scale factor
IMAGESWV= 'CFHT DetCom v3.60.18 (Apr 20 2017)' / Image creation software version
OBSTYPE = 'OBJECT ' / Observation / Exposure type
EXPTYPE = 'OBJECT ' / See OBSTYPE
EXPTIME = 2561.0 / Integration time (seconds)
DARKTIME= 2561.0 / Dark current time (seconds)
DETECTOR= 'OLAPA ' / Science Detector
CCD = 'Unknown ' / Science Detector (use DETECTOR)
IMAGEID = 0 / CCD chip number
CHIPID = 0 / Use IMAGEID instead
DETSIZE = '[1:2048,1:4608]' / Total data pixels in full mosaic
RASTER = 'FULL ' / Active raster description
CCDSUM = '1 1 ' / Binning factors
CCDBIN1 = 1 / Binning factor along first axis
CCDBIN2 = 1 / Binning factor along second axis
PIXSIZE = 13.5 / Pixel size for both axes (microns)
AMPLIST = 'a,b ' / List of amplifiers for this image
CCDSIZE = '[1:2048,1:4608]' / Detector imaging area size
CCDSEC = '[21:2068,1:4608]' / Read out area of the detector (unbinned)
TRIMSEC = '[21:2068,4:4605]' / Useful imaging area of the detector
BSECA = '[1:20,1:4608]' / Overscan/prescan (bias) area from Amp A
BSECB = '[2069:2088,1:4608]' / Overscan/prescan (bias) area from Amp B
CSECA = '[21:1044,1:4608]' / Section in full CCD for DSECA
CSECB = '[1045:2068,1:4608]' / Section in full CCD for DSECB
DSECA = '[21:1044,1:4608]' / Imaging area from Amp A
DSECB = '[1045:2068,1:4608]' / Imaging area from Amp B
TSECA = '[21:1044,4:4605]' / Trim section for Amp A
TSECB = '[1045:2068,4:4605]' / Trim section for Amp B
MAXLIN = 65535 / Maximum linearity value (ADU)
SATURATE= 65535 / Saturation value (ADU)
GAINA = 1.10 / Amp A gain (electrons/ADU)
GAINB = 1.20 / Amp B gain (electrons/ADU)
RDNOISEA= 2.90 / Amp A read noise (electrons)
RDNOISEB= 2.90 / Amp B read noise (electrons)
DARKCUR = 0 / Dark current (e-/pixel/hour)
RDTIME = 30.00 / Read out time (sec)
CONSWV = 'olD=137,DCU=49' / Controller software DSPID and SERNO versions
DETSTAT = 'ok ' / Detector temp range (-105..-90)
DETTEM = -100.2 / Detector temp deg C = 745.502 + -0.278 * 3042
INSTRUME= 'GRACES ' / Instrument Name
ECAMFOC = -3.61 / ESPaDOnS camera focus position (mm)
EHARTPOS= 'OUT ' / ESPaDOnS hartmann position FULL/DOWN/UP/OUT
EEMSHUT = 'CLOSE ' / ESPaDOnS exposure meter shutter OPEN/CLOSED
EEMSTATE= 'OFF ' / ESPaDOnS exposure meter state ON/OFF
EEMCNTS = -9999 / ESPaDOnS exposure meter count average
ETSP1BEG= 17.38 / ESPaDOnS down mirror temp at start (deg C)
ETSP2BEG= 17.61 / ESPaDOnS camera temp at start (deg C)
ETSP3BEG= 17.55 / ESPaDOnS up mirror temp at start (deg C)
ETSP4BEG= 17.31 / ESPaDOnS hygrometer temp at start (deg C)
EPRSPBEG= -3.35 / ESPaDOnS relative pressure at start (mb)
ERHSPBEG= 21.11 / ESPaDOnS relative humidity at start (%)
ETSP1END= 17.38 / ESPaDOnS down mirror temp at end (deg C)
ETSP2END= 17.59 / ESPaDOnS camera temp at end (deg C)
ETSP3END= 17.55 / ESPaDOnS up mirror temp at end (deg C)
ETSP4END= 17.31 / ESPaDOnS hygrometer temp at end (deg C)
EPRSPEND= -3.29 / ESPaDOnS relative pressure at end (mb)
EREADSPD= 'Slow: 2.90e noise, 1.15e/ADU, 30s' / ESPaDOnS det read out xslow/slow
GSLIPOS = 'TWOSLICE' / GRACES slicer bench position (# and mm)
GSLICER = 'TWOSLICE' / GRACES slicer position (# and deg)
GDEKKER = 'TWOSLICE' / GRACES dekker position (# and mm)
GPMIRROR= 'GEMINI ' / GRACES pickoff mirror position (# and mm)
GFIBMODE= 'GRACES ' / GRACES fiber position (ESPADONS or GRACES)
O_BSCALE= 1.00000 / Original BSCALE Value
RAWIQ = '85-percentile' /Raw Image Quality
RAWCC = '50-percentile' /Raw Cloud Cover
RAWWV = '20-percentile' /Raw Water Vapour/Transparency
RAWBG = '50-percentile' /Raw Background
TELESCOP= 'Gemini-North' /Gemini-North
EPOCH = 2000.00 /Epoch for Target coordinates
CRPA = 80.7411581783 /Current Cass Rotator Position Angle
AIRMASS = '1.271 ' /Mean airmass for the observation
AMSTART = '1.365 ' /Airmass at start of exposure
AMEND = '1.191 ' /Airmass at end of exposure
HA = '-03:02:41.87' /Hour Angle Sexagesimal
HAD = '-3.0449640' /Hour Angle Decimal
OBSCLASS= 'science ' /Observe class
INSTMODE= 'Spectroscopy, star+sky' /Observing mode
RAWPIREQ= 'YES ' /PI Requirements Met
RAWGEMQA= 'USABLE ' /Gemini Quality Assessment
COMMENT ----------------------------------------------------
COMMENT | Processed by the CFHT OPERA Open Source Pipeline |
COMMENT ----------------------------------------------------
COMMENT opera-1.0.1228 build date Fri May 19 18:53:30 HST 2017
COMMENT Processing Date
COMMENT ---------------
COMMENT Mon May 4 14:50:27 HST 2020
COMMENT ------------------------------------------------------------------------
COMMENT 20
SNR22 = '0.50798 / 0.61051' / snr per spectral / ccd bin
SNR23 = '0.84309 / 1.0133' / snr per spectral / ccd bin
SNR24 = '1.2171 / 1.4627' / snr per spectral / ccd bin
SNR25 = '1.9608 / 2.3567' / snr per spectral / ccd bin
SNR26 = '2.1154 / 2.5425' / snr per spectral / ccd bin
SNR27 = '2.1107 / 2.5368' / snr per spectral / ccd bin
SNR28 = '2.2236 / 2.6725' / snr per spectral / ccd bin
SNR29 = '2.1241 / 2.5528' / snr per spectral / ccd bin
SNR30 = '2.2457 / 2.6991' / snr per spectral / ccd bin
SNR31 = '1.9948 / 2.3975' / snr per spectral / ccd bin
SNR32 = '1.6974 / 2.04' / snr per spectral / ccd bin
SNR33 = '1.4978 / 1.8001' / snr per spectral / ccd bin
SNR34 = '1.2949 / 1.5562' / snr per spectral / ccd bin
SNR35 = '1.3562 / 1.63' / snr per spectral / ccd bin
SNR36 = '1.0198 / 1.2257' / snr per spectral / ccd bin
SNR37 = '1.5021 / 1.8053' / snr per spectral / ccd bin
SNR38 = '1.01 / 1.2139' / snr per spectral / ccd bin
SNR39 = '0.71061 / 0.85405' / snr per spectral / ccd bin
SNR40 = '0.59577 / 0.71603' / snr per spectral / ccd bin
SNR41 = '0.62022 / 0.74541' / snr per spectral / ccd bin
SNR42 = '0.57458 / 0.69056' / snr per spectral / ccd bin
SNR43 = '0.52749 / 0.63397' / snr per spectral / ccd bin
SNR44 = '0.49149 / 0.5907' / snr per spectral / ccd bin
SNR45 = '0.48021 / 0.57714' / snr per spectral / ccd bin
SNR46 = '0.50494 / 0.60687' / snr per spectral / ccd bin
SNR47 = '0.52551 / 0.63159' / snr per spectral / ccd bin
SNR48 = '0.5106 / 0.61367' / snr per spectral / ccd bin
SNR49 = '0.42632 / 0.51237' / snr per spectral / ccd bin
SNR50 = '0.42451 / 0.5102' / snr per spectral / ccd bin
SNR51 = '0.42025 / 0.50508' / snr per spectral / ccd bin
SNR52 = '0.41137 / 0.49441' / snr per spectral / ccd bin
SNR53 = '0.41708 / 0.50127' / snr per spectral / ccd bin
SNR54 = '0.41904 / 0.50362' / snr per spectral / ccd bin
SNR55 = '0.4284 / 0.51487' / snr per spectral / ccd bin
HRV = -8.9311 / Heliocentric RV correction (km/s)
HRVLUNAR= 0.0125599 / lunar component of HRV correction (km/s)
HRVORBIT= -9.19911 / orbital component of HRV correction (km/s)
HRVDIURN= 0.255451 / diurnal component of HRV correction (km/s)
HJDUTC = 2458568.79603 / Heliocentric Julian date (UTC) mid-exposure
HJDTT = 2458568.796831 / Heliocentric Julian date (TT) mid-exposure
TELLRV = 0. / telluric RV correction (km/s)
TELLERR = 0. / telluric RV correction error (km/s)
REDUCTIO= 'Intensity' / Type of reduction
NORMAL = '2 ' / Normalized and Un-normalized Data
COMMENT File contains automatic wavelength correction and uncorrected data.
COL1 = 'Wavelength' / Normalized
COL2 = 'Star ' / Normalized
COL3 = 'Sky ' / Normalized
COL4 = 'Star+sky' / Normalized
COL5 = 'ErrorBarStar' / Normalized
COL6 = 'ErrorBarSky' / Normalized
COL7 = 'ErrorBarStar+Sky' / Normalized
COL8 = 'Wavelength' / UnNormalized
COL9 = 'Star ' / UnNormalized
COL10 = 'Sky ' / UnNormalized
COL11 = 'Star+sky' / UnNormalized
COL12 = 'ErrorBarStar' / UnNormalized
COL13 = 'ErrorBarSky' / UnNormalized
COL14 = 'ErrorBarStar+Sky' / UnNormalized
COL15 = 'Wavelength' / Normalized, no autowave correction
COL16 = 'Star ' / Normalized, no autowave correction
COL17 = 'Sky ' / Normalized, no autowave correction
COL18 = 'Star+sky' / Normalized, no autowave correction
COL19 = 'ErrorBarStar' / Normalized, no autowave correction
COL20 = 'ErrorBarSky' / Normalized, no autowave correction
COL21 = 'ErrorBarStar+Sky' / Normalized, no autowave correction
COL22 = 'Wavelength' / UnNormalized, no autowave correction
COL23 = 'Star ' / UnNormalized, no autowave correction
COL24 = 'Sky ' / UnNormalized, no autowave correction
COL25 = 'Star+sky' / UnNormalized, no autowave correction
COL26 = 'ErrorBarStar' / UnNormalized, no autowave correction
COL27 = 'ErrorBarSky' / UnNormalized, no autowave correction
COL28 = 'ErrorBarStar+Sky' / UnNormalized, no autowave correction

Answer 1

Since they seemed to help I'm rewriting my above comments as an answer:
The data in FITS files (e.g. hdulist[0].data) are returned as Numpy arrays. Numpy is a core library to many scientific Python packages for working with binary array-like data. This is good to be aware of because it's not specific to Astropy or FITS, and any question about how to work with data from FITS files (that isn't Astronomy-specific) is really a question about Numpy. Numpy has a built-in function np.savetxt for this purpose. E.g.
>>> import numpy as np
>>> np.savetxt(hdulist[0].data)
np.savetxt takes numerous options you can read about in the API documentation for how to format your data.
(Side question: Why do you want to save it as a plain text file? It seems like an fairly large array--is there something you want to do with it as a text file that you can't in binary?)
Second, are a couple reasons your attempts to use Table.read failed. For one, your data does not appear to be tabular, so this wouldn't be appropriate for CCD data in this form. Second Table.read is a more abstract function that just takes the name of a file (or a "file-like object, meaning something that has the same interface of the file objects returned from Python's built-in open function). It automatically guesses how to read the tabular data by recognizing some supported file formats. You were passing it objects it doesn't know what to do with, hence getting seemingly obscure errors.
For example Table.read is used like this:
>>> Table.read('path/to/fits/file/containing/table.fits')
Under the hood this is using the astropy.io.fits package to parse the FITS file and extract table contents. The interface is designed to abstract away those details from the user. In your first attempt you passed it an actual HDU object (hdulist[0]). Since you specified format='fits' it tries to use its FITS reader on this object, but it doesn't know what to do with it because it's not a filename or a file-like object. Similar problem with your second attempt.
Finally for the reason this didn't work as you were hoping:
>>> tbdata = fits.getdata('N20190326G0041i.fits')
>>> print(tbdata,file=open('test.txt','a'))
This is no different from print(tbdata). When you print a Numpy array to the screen it has a standard print representation which for large arrays normally truncates the data. Using file= doesn't do anything magic: It just outputs the same thing you would get printing the array to the terminal, but it outputs that text to a file instead of the screen.