Can't print entire data set with fits from astropy.io - python-3.x

I have a large fits file (21.4 MB). I would like to print the contents of it to a text file, but can only access a portion of it. I am looking for help getting the entire file to text format.
> from astropy.io import fits
> hdulist = fits.open('N20190326G0041i.fits')
Information on the file. Note that everything is in the primary HDU.
> hdulist.info()
Filename: N20190326G0041i.fits
No. Name Ver Type Cards Dimensions Format
0 PRIMARY 1 PrimaryHDU 183 (190685, 28) float32
I can access the full header but it is extremely long. I included it at the end of this post.
> hdu = hdulist[0]
> hdu.header
However, I only get a portion of the data using hdu.data
> hdu.data
array([[ 4.0630740e+02, 4.0631021e+02, 4.0631290e+02, ...,
1.0478779e+03, 1.0478831e+03, 1.0478882e+03],
[ 2.7955999e+01, 3.1493999e+01, 1.2378000e+01, ...,
-4.3614998e+00, -1.8785000e+00, -8.8672000e-01],
[ 2.8534999e+00, 2.8862000e+00, 2.9282999e+00, ...,
-6.1020999e+00, -5.2989998e+00, -5.1680999e+00],
...,
[ 1.7951000e+04, 2.9099000e+04, 3.5257000e+03, ...,
1.0594000e+03, 7.9347998e+02, 1.6349001e+02],
[ 3.1568999e+03, 3.1631001e+03, 3.2426001e+03, ...,
3.2828000e+02, 3.2062000e+02, 3.2189001e+02],
[ 3.3338000e+03, 3.3806001e+03, 3.4557000e+03, ...,
2.1803000e+02, 2.2574001e+02, 2.3003999e+02]], dtype=float32)
What I typically do to print fits files to text files is ...
> table = hdulist[0].data
> print(table, file = open('test.txt','a'))
This "works", and outputs the same excerpt of the data that hdu.data prints on screen.
> [[ 4.0630740e+02 4.0631021e+02 4.0631290e+02 ... 1.0478779e+03
1.0478831e+03 1.0478882e+03]
[ 2.7955999e+01 3.1493999e+01 1.2378000e+01 ... -4.3614998e+00
-1.8785000e+00 -8.8672000e-01]
[ 2.8534999e+00 2.8862000e+00 2.9282999e+00 ... -6.1020999e+00
-5.2989998e+00 -5.1680999e+00]
...
[ 1.7951000e+04 2.9099000e+04 3.5257000e+03 ... 1.0594000e+03
7.9347998e+02 1.6349001e+02]
[ 3.1568999e+03 3.1631001e+03 3.2426001e+03 ... 3.2828000e+02
3.2062000e+02 3.2189001e+02]
[ 3.3338000e+03 3.3806001e+03 3.4557000e+03 ... 2.1803000e+02
2.2574001e+02 2.3003999e+02]]
Also, I repeated all of the above things using memmap = True, but get the same results.
> from astropy.io import fits
> hdulist = fits.open('N20190326G0041i.fits', memmap = True)
I also tried the convenience functions, but that produced the exact same excerpt as hdu.data.
> tbdata = fits.getdata('N20190326G0041i.fits')
> print(tbdata,file=open('test.txt','a'))
I also tried the astropy.table package, but could not get it to work either.
> from astropy.table import Table
> t = Table.read(hdulist[0], format = 'fits')
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/anaconda3/lib/python3.7/site-packages/astropy/table/connect.py", line 52, in __call__
out = registry.read(cls, *args, **kwargs)
File "/anaconda3/lib/python3.7/site-packages/astropy/io/registry.py", line 523, in read
data = reader(*args, **kwargs)
File "/anaconda3/lib/python3.7/site-packages/astropy/io/fits/connect.py", line 195, in read_table_fits
memmap=memmap)
File "/anaconda3/lib/python3.7/site-packages/astropy/io/fits/hdu/hdulist.py", line 151, in fitsopen
lazy_load_hdus, **kwargs)
File "/anaconda3/lib/python3.7/site-packages/astropy/io/fits/hdu/hdulist.py", line 390, in fromfile
lazy_load_hdus=lazy_load_hdus, **kwargs)
File "/anaconda3/lib/python3.7/site-packages/astropy/io/fits/hdu/hdulist.py", line 1039, in _readfrom
fileobj = _File(fileobj, mode=mode, memmap=memmap, cache=cache)
File "/anaconda3/lib/python3.7/site-packages/astropy/utils/decorators.py", line 521, in wrapper
return function(*args, **kwargs)
File "/anaconda3/lib/python3.7/site-packages/astropy/io/fits/file.py", line 180, in __init__
self._open_filelike(fileobj, mode, overwrite)
File "/anaconda3/lib/python3.7/site-packages/astropy/io/fits/file.py", line 533, in _open_filelike
"method, required for mode '{}'.".format(self.mode))
OSError: File-like object does not have a 'write' method, required for mode 'ostream'.
However, if I use
> t=Table.read(hdu.data, format='fits')
then I get a different error
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/anaconda3/lib/python3.7/site-packages/astropy/table/connect.py", line 52, in __call__
out = registry.read(cls, *args, **kwargs)
File "/anaconda3/lib/python3.7/site-packages/astropy/io/registry.py", line 523, in read
data = reader(*args, **kwargs)
File "/anaconda3/lib/python3.7/site-packages/astropy/io/fits/connect.py", line 195, in read_table_fits
memmap=memmap)
File "/anaconda3/lib/python3.7/site-packages/astropy/io/fits/hdu/hdulist.py", line 147, in fitsopen
if not name:
ValueError: The truth value of an array with more than one element is ambiguous. Use a.any() or a.all()
I have also tried tprint and tdump in PyRAF, it simply gives the error "Warning: ", with no other helpful information. I also tried writing the fits file to text using the wspectext function in PyRaf, but it is unable to do so.
However, I want to be able to just use the fits package from astropy to output the data to a text file. I have done this countless times with other fits files from a variety of telescope pipelines, but it isn't working this time. Any help is much appreciated.
The header of the file is below. I'm wondering since the header was written in IDL if the only way I'll be able to access the info is through IDL? This seems unlikely to me. I'd really rather avoid using IDL if possible. We have a site license at our university, but we are not permitted on campus during the pandemic and our VPN capabilities are lacking.
> hdu.header
SIMPLE = T / Written by IDL:
BITPIX = -32 / Real*4 (complex, stored as float)
NAXIS = 2 / Number of axes
NAXIS1 = 190685 / Number of pixel columns
NAXIS2 = 28 / Number of pixel rows
INHERIT = F / No need to inherit global keywords
BZERO = 0. / Data is Unsigned Integer
BSCALE = 1. / Scale factor
IMAGESWV= 'CFHT DetCom v3.60.18 (Apr 20 2017)' / Image creation software version
OBSTYPE = 'OBJECT ' / Observation / Exposure type
EXPTYPE = 'OBJECT ' / See OBSTYPE
EXPTIME = 2561.0 / Integration time (seconds)
DARKTIME= 2561.0 / Dark current time (seconds)
DETECTOR= 'OLAPA ' / Science Detector
CCD = 'Unknown ' / Science Detector (use DETECTOR)
IMAGEID = 0 / CCD chip number
CHIPID = 0 / Use IMAGEID instead
DETSIZE = '[1:2048,1:4608]' / Total data pixels in full mosaic
RASTER = 'FULL ' / Active raster description
CCDSUM = '1 1 ' / Binning factors
CCDBIN1 = 1 / Binning factor along first axis
CCDBIN2 = 1 / Binning factor along second axis
PIXSIZE = 13.5 / Pixel size for both axes (microns)
AMPLIST = 'a,b ' / List of amplifiers for this image
CCDSIZE = '[1:2048,1:4608]' / Detector imaging area size
CCDSEC = '[21:2068,1:4608]' / Read out area of the detector (unbinned)
TRIMSEC = '[21:2068,4:4605]' / Useful imaging area of the detector
BSECA = '[1:20,1:4608]' / Overscan/prescan (bias) area from Amp A
BSECB = '[2069:2088,1:4608]' / Overscan/prescan (bias) area from Amp B
CSECA = '[21:1044,1:4608]' / Section in full CCD for DSECA
CSECB = '[1045:2068,1:4608]' / Section in full CCD for DSECB
DSECA = '[21:1044,1:4608]' / Imaging area from Amp A
DSECB = '[1045:2068,1:4608]' / Imaging area from Amp B
TSECA = '[21:1044,4:4605]' / Trim section for Amp A
TSECB = '[1045:2068,4:4605]' / Trim section for Amp B
MAXLIN = 65535 / Maximum linearity value (ADU)
SATURATE= 65535 / Saturation value (ADU)
GAINA = 1.10 / Amp A gain (electrons/ADU)
GAINB = 1.20 / Amp B gain (electrons/ADU)
RDNOISEA= 2.90 / Amp A read noise (electrons)
RDNOISEB= 2.90 / Amp B read noise (electrons)
DARKCUR = 0 / Dark current (e-/pixel/hour)
RDTIME = 30.00 / Read out time (sec)
CONSWV = 'olD=137,DCU=49' / Controller software DSPID and SERNO versions
DETSTAT = 'ok ' / Detector temp range (-105..-90)
DETTEM = -100.2 / Detector temp deg C = 745.502 + -0.278 * 3042
INSTRUME= 'GRACES ' / Instrument Name
ECAMFOC = -3.61 / ESPaDOnS camera focus position (mm)
EHARTPOS= 'OUT ' / ESPaDOnS hartmann position FULL/DOWN/UP/OUT
EEMSHUT = 'CLOSE ' / ESPaDOnS exposure meter shutter OPEN/CLOSED
EEMSTATE= 'OFF ' / ESPaDOnS exposure meter state ON/OFF
EEMCNTS = -9999 / ESPaDOnS exposure meter count average
ETSP1BEG= 17.38 / ESPaDOnS down mirror temp at start (deg C)
ETSP2BEG= 17.61 / ESPaDOnS camera temp at start (deg C)
ETSP3BEG= 17.55 / ESPaDOnS up mirror temp at start (deg C)
ETSP4BEG= 17.31 / ESPaDOnS hygrometer temp at start (deg C)
EPRSPBEG= -3.35 / ESPaDOnS relative pressure at start (mb)
ERHSPBEG= 21.11 / ESPaDOnS relative humidity at start (%)
ETSP1END= 17.38 / ESPaDOnS down mirror temp at end (deg C)
ETSP2END= 17.59 / ESPaDOnS camera temp at end (deg C)
ETSP3END= 17.55 / ESPaDOnS up mirror temp at end (deg C)
ETSP4END= 17.31 / ESPaDOnS hygrometer temp at end (deg C)
EPRSPEND= -3.29 / ESPaDOnS relative pressure at end (mb)
EREADSPD= 'Slow: 2.90e noise, 1.15e/ADU, 30s' / ESPaDOnS det read out xslow/slow
GSLIPOS = 'TWOSLICE' / GRACES slicer bench position (# and mm)
GSLICER = 'TWOSLICE' / GRACES slicer position (# and deg)
GDEKKER = 'TWOSLICE' / GRACES dekker position (# and mm)
GPMIRROR= 'GEMINI ' / GRACES pickoff mirror position (# and mm)
GFIBMODE= 'GRACES ' / GRACES fiber position (ESPADONS or GRACES)
O_BSCALE= 1.00000 / Original BSCALE Value
RAWIQ = '85-percentile' /Raw Image Quality
RAWCC = '50-percentile' /Raw Cloud Cover
RAWWV = '20-percentile' /Raw Water Vapour/Transparency
RAWBG = '50-percentile' /Raw Background
TELESCOP= 'Gemini-North' /Gemini-North
EPOCH = 2000.00 /Epoch for Target coordinates
CRPA = 80.7411581783 /Current Cass Rotator Position Angle
AIRMASS = '1.271 ' /Mean airmass for the observation
AMSTART = '1.365 ' /Airmass at start of exposure
AMEND = '1.191 ' /Airmass at end of exposure
HA = '-03:02:41.87' /Hour Angle Sexagesimal
HAD = '-3.0449640' /Hour Angle Decimal
OBSCLASS= 'science ' /Observe class
INSTMODE= 'Spectroscopy, star+sky' /Observing mode
RAWPIREQ= 'YES ' /PI Requirements Met
RAWGEMQA= 'USABLE ' /Gemini Quality Assessment
COMMENT ----------------------------------------------------
COMMENT | Processed by the CFHT OPERA Open Source Pipeline |
COMMENT ----------------------------------------------------
COMMENT opera-1.0.1228 build date Fri May 19 18:53:30 HST 2017
COMMENT Processing Date
COMMENT ---------------
COMMENT Mon May 4 14:50:27 HST 2020
COMMENT ------------------------------------------------------------------------
COMMENT 20
SNR22 = '0.50798 / 0.61051' / snr per spectral / ccd bin
SNR23 = '0.84309 / 1.0133' / snr per spectral / ccd bin
SNR24 = '1.2171 / 1.4627' / snr per spectral / ccd bin
SNR25 = '1.9608 / 2.3567' / snr per spectral / ccd bin
SNR26 = '2.1154 / 2.5425' / snr per spectral / ccd bin
SNR27 = '2.1107 / 2.5368' / snr per spectral / ccd bin
SNR28 = '2.2236 / 2.6725' / snr per spectral / ccd bin
SNR29 = '2.1241 / 2.5528' / snr per spectral / ccd bin
SNR30 = '2.2457 / 2.6991' / snr per spectral / ccd bin
SNR31 = '1.9948 / 2.3975' / snr per spectral / ccd bin
SNR32 = '1.6974 / 2.04' / snr per spectral / ccd bin
SNR33 = '1.4978 / 1.8001' / snr per spectral / ccd bin
SNR34 = '1.2949 / 1.5562' / snr per spectral / ccd bin
SNR35 = '1.3562 / 1.63' / snr per spectral / ccd bin
SNR36 = '1.0198 / 1.2257' / snr per spectral / ccd bin
SNR37 = '1.5021 / 1.8053' / snr per spectral / ccd bin
SNR38 = '1.01 / 1.2139' / snr per spectral / ccd bin
SNR39 = '0.71061 / 0.85405' / snr per spectral / ccd bin
SNR40 = '0.59577 / 0.71603' / snr per spectral / ccd bin
SNR41 = '0.62022 / 0.74541' / snr per spectral / ccd bin
SNR42 = '0.57458 / 0.69056' / snr per spectral / ccd bin
SNR43 = '0.52749 / 0.63397' / snr per spectral / ccd bin
SNR44 = '0.49149 / 0.5907' / snr per spectral / ccd bin
SNR45 = '0.48021 / 0.57714' / snr per spectral / ccd bin
SNR46 = '0.50494 / 0.60687' / snr per spectral / ccd bin
SNR47 = '0.52551 / 0.63159' / snr per spectral / ccd bin
SNR48 = '0.5106 / 0.61367' / snr per spectral / ccd bin
SNR49 = '0.42632 / 0.51237' / snr per spectral / ccd bin
SNR50 = '0.42451 / 0.5102' / snr per spectral / ccd bin
SNR51 = '0.42025 / 0.50508' / snr per spectral / ccd bin
SNR52 = '0.41137 / 0.49441' / snr per spectral / ccd bin
SNR53 = '0.41708 / 0.50127' / snr per spectral / ccd bin
SNR54 = '0.41904 / 0.50362' / snr per spectral / ccd bin
SNR55 = '0.4284 / 0.51487' / snr per spectral / ccd bin
HRV = -8.9311 / Heliocentric RV correction (km/s)
HRVLUNAR= 0.0125599 / lunar component of HRV correction (km/s)
HRVORBIT= -9.19911 / orbital component of HRV correction (km/s)
HRVDIURN= 0.255451 / diurnal component of HRV correction (km/s)
HJDUTC = 2458568.79603 / Heliocentric Julian date (UTC) mid-exposure
HJDTT = 2458568.796831 / Heliocentric Julian date (TT) mid-exposure
TELLRV = 0. / telluric RV correction (km/s)
TELLERR = 0. / telluric RV correction error (km/s)
REDUCTIO= 'Intensity' / Type of reduction
NORMAL = '2 ' / Normalized and Un-normalized Data
COMMENT File contains automatic wavelength correction and uncorrected data.
COL1 = 'Wavelength' / Normalized
COL2 = 'Star ' / Normalized
COL3 = 'Sky ' / Normalized
COL4 = 'Star+sky' / Normalized
COL5 = 'ErrorBarStar' / Normalized
COL6 = 'ErrorBarSky' / Normalized
COL7 = 'ErrorBarStar+Sky' / Normalized
COL8 = 'Wavelength' / UnNormalized
COL9 = 'Star ' / UnNormalized
COL10 = 'Sky ' / UnNormalized
COL11 = 'Star+sky' / UnNormalized
COL12 = 'ErrorBarStar' / UnNormalized
COL13 = 'ErrorBarSky' / UnNormalized
COL14 = 'ErrorBarStar+Sky' / UnNormalized
COL15 = 'Wavelength' / Normalized, no autowave correction
COL16 = 'Star ' / Normalized, no autowave correction
COL17 = 'Sky ' / Normalized, no autowave correction
COL18 = 'Star+sky' / Normalized, no autowave correction
COL19 = 'ErrorBarStar' / Normalized, no autowave correction
COL20 = 'ErrorBarSky' / Normalized, no autowave correction
COL21 = 'ErrorBarStar+Sky' / Normalized, no autowave correction
COL22 = 'Wavelength' / UnNormalized, no autowave correction
COL23 = 'Star ' / UnNormalized, no autowave correction
COL24 = 'Sky ' / UnNormalized, no autowave correction
COL25 = 'Star+sky' / UnNormalized, no autowave correction
COL26 = 'ErrorBarStar' / UnNormalized, no autowave correction
COL27 = 'ErrorBarSky' / UnNormalized, no autowave correction
COL28 = 'ErrorBarStar+Sky' / UnNormalized, no autowave correction

Since they seemed to help I'm rewriting my above comments as an answer:
The data in FITS files (e.g. hdulist[0].data) are returned as Numpy arrays. Numpy is a core library to many scientific Python packages for working with binary array-like data. This is good to be aware of because it's not specific to Astropy or FITS, and any question about how to work with data from FITS files (that isn't Astronomy-specific) is really a question about Numpy. Numpy has a built-in function np.savetxt for this purpose. E.g.
>>> import numpy as np
>>> np.savetxt(hdulist[0].data)
np.savetxt takes numerous options you can read about in the API documentation for how to format your data.
(Side question: Why do you want to save it as a plain text file? It seems like an fairly large array--is there something you want to do with it as a text file that you can't in binary?)
Second, are a couple reasons your attempts to use Table.read failed. For one, your data does not appear to be tabular, so this wouldn't be appropriate for CCD data in this form. Second Table.read is a more abstract function that just takes the name of a file (or a "file-like object, meaning something that has the same interface of the file objects returned from Python's built-in open function). It automatically guesses how to read the tabular data by recognizing some supported file formats. You were passing it objects it doesn't know what to do with, hence getting seemingly obscure errors.
For example Table.read is used like this:
>>> Table.read('path/to/fits/file/containing/table.fits')
Under the hood this is using the astropy.io.fits package to parse the FITS file and extract table contents. The interface is designed to abstract away those details from the user. In your first attempt you passed it an actual HDU object (hdulist[0]). Since you specified format='fits' it tries to use its FITS reader on this object, but it doesn't know what to do with it because it's not a filename or a file-like object. Similar problem with your second attempt.
Finally for the reason this didn't work as you were hoping:
>>> tbdata = fits.getdata('N20190326G0041i.fits')
>>> print(tbdata,file=open('test.txt','a'))
This is no different from print(tbdata). When you print a Numpy array to the screen it has a standard print representation which for large arrays normally truncates the data. Using file= doesn't do anything magic: It just outputs the same thing you would get printing the array to the terminal, but it outputs that text to a file instead of the screen.

Related

How to count how many instances of a list's items occur in a column with multiple string entries per row?

I have a dataframe column signified df['Q2'] with all the responses from a users survey for the question 'which indicators do you use'? There are 36 statistical indicators users can choose from and one additional answer which is simply 'All of the indicators'. There are multiple string answers in each row as an artefact of the survey designed by someone else.
How do I simply pattern match/cross refer between respondents' answers of multiple indicators across multiple lines in the rows of column 2, with the indicators in my neat list?
indicators = ["All of the indicators",
"A1 / Eng13",
"A2 / Eng14",
"Eng14b",
"A4 / Eng17",
"A5",
"Eng16",
"B1a / Eng22a",
"B1b / Eng22b",
"B2 / Eng23",
"B4 / Eng18",
"B5a / Eng19a",
"B5b / Eng19b",
"B6 / Eng20",
"B7 / Eng21",
"C1 / Eng1",
"C2 / Eng3",
"Eng2a",
"C3a / Eng2b",
"C3b / Eng4c",
"C4a / Eng4a",
"C4b / Eng4b",
"C5",
"C6",
"C7",
"C8",
"Eng5",
"Eng6",
"Eng7",
"Eng8",
"C9a / Eng12a",
"C9b / Eng12b",
"D1a / Eng11",
"D1b / Eng9",
"D1c / Eng10",
"E1 / Eng24",
"E2 / Eng15"]
indicators_mentioned = df['Q2'] #all the responses in column 2 of dataframe
indicators_mentioned_as_string = indicators_mentioned.to_string() # convert responses to string
And regex/pattern matching attempt:
indicator_regex = re.compile(r'All of the indicators') #set a pattern to match for each indicator
instances_of_all_indicators = indicator_regex.findall(indicators_mentioned_as_string)
#assigns a variable to store all instances of indicators returned by find all function which finds all #instances within the bracketed dataset
instances_of_all_indicators_summed = (len(instances_of_all_indicators)) #list that is 6 items long contains 6 instances!
type(instances_of_all_indicators_summed) #check the value in the variable is int to go into the dictionary below as value
print(instances_of_all_indicators_summed)
#regex for second indicator
indicator_regex = re.compile(r'A1 / Eng13')
instances_of_A1 = indicator_regex.findall(indicators_mentioned_as_string)
instances_of_A1_summed = len(instances_of_A1)
print(instances_of_A1_summed)
I would like all the responses per indicator to go into a dictionary from which I can then make a nice chart.
indicator_by_reponse = {
"All of the indicators": instances_of_all_indicators_summed,
"A1 / Eng13": instances_of_A1_summed,
"A2 / Eng14": instances_of_A2_summed,
"Eng14b": instances_of_Eng14b_summed,
# "A4 / Eng17": instances_of_A4_summed,
# "A5": instances_of_A5_summed,
# "Eng16": instances_of_Eng16_summed,
# "B1a / Eng22a": instances_of_B1a_summed,
# "B1b / Eng22b": instances_of_B1b_summed,
# "B2 / Eng23": instances_of_B2_summed,
# "B4 / Eng18": instances_of_B4_summed,
# "B5a / Eng19a": instances_of_B5a_summed,
# "B5b / Eng19b": instances_of_B5b_summed,
# "B6 / Eng20": instances_of_B6_summed,
# "B7 / Eng21": instances_of_B7_summed,
# "C1 / Eng1": instances_of_C1_summed,
# "C2 / Eng3": instances_of_C2_summed,
# "Eng2a": instances_of_Eng2a_summed,
# "C3a / Eng2b": instances_of_C3a_summed,
# "C3b / Eng4c": instances_of_C3b_summed,
# "C4a / Eng4a": instances_of_C4a_summed,
# "C4b / Eng4b": instances_of_C4b_summed,
# "C5": instances_of_C5_summed,
# "C6": instances_of_C6_summed,
# "C7": instances_of_C7_summed,
# "C8": instances_of_C8_summed,
# "Eng5": instances_of_Eng5_summed,
# "Eng6": instances_of_Eng6_summed,
# "Eng7": instances_of_Eng7_summed,
# "Eng8": instances_of_Eng8_summed,
# "C9a / Eng12a": instances_of_C9a_summed,
# "C9b / Eng12b": instances_of_C9b_summed,
# "D1a / Eng11": instances_of_D1a_summed,
# "D1b / Eng9": instances_of_D1b_summed,
# "D1c / Eng10": instances_of_D1c_summed,
# "E1 / Eng24": instances_of_E1_summed,
# "E2 / Eng15": instances_of_E2_summed,
}
I converted my pandas series to a string and then used findall() but that still returns the responses in a messy way when what I want to do is group instances of the string for each indicator in the list to make a neat dictionary and a chart. To pattern match I used regex after importing re but this is a completely inelegant way of doing it which requires me to manually count the responses to put into a dictionary; there must be a simpler way.

Pan Tompkins Lowpass filter overflow

The Pan Tompkins algorithm1 for removing noise from an ECG/EKG is cited often. They use a low pass filter, followed by a high pass filter. The output of the high pass filter looks great. But (depending on starting conditions) the output of the low pass filter will continuously increase or decrease. Given enough time, your numbers will eventually get to a size that the programming language cannot handle and rollover. If I run this on an Arduino (which uses a variant of C), it rolls over on the order of 10 seconds. Not ideal. Is there a way to get rid of this bias? I've tried messing with initial conditions, but I'm fresh out of ideas. The advantage of this algorithm is that it's not very computationally intensive and will run comfortably on a modest microprocessor.
1 Pan, Jiapu; Tompkins, Willis J. (March 1985). "A Real-Time QRS Detection Algorithm". IEEE Transactions on Biomedical Engineering. BME-32 (3): 230–236.
Python code to illustrate problem. Uses numpy and matplotlib:
import numpy as np
import matplotlib.pyplot as plt
#low-pass filter
def lpf(x):
y = x.copy()
for n in range(len(x)):
if(n < 12):
continue
y[n,1] = 2*y[n-1,1] - y[n-2,1] + x[n,1] - 2*x[n-6,1] + x[n-12,1]
return y
#high-pass filter
def hpf(x):
y = x.copy()
for n in range(len(x)):
if(n < 32):
continue
y[n,1] = y[n-1,1] - x[n,1]/32 + x[n-16,1] - x[n-17,1] + x[n-32,1]/32
return y
ecg = np.loadtxt('ecg_data.csv', delimiter=',',skiprows=1)
plt.plot(ecg[:,0], ecg[:,1])
plt.title('Raw Data')
plt.grid(True)
plt.savefig('raw.png')
plt.show()
#Application of lpf
f1 = lpf(ecg)
plt.plot(f1[:,0], f1[:,1])
plt.title('After Pan-Tompkins LPF')
plt.xlabel('time')
plt.ylabel('mV')
plt.grid(True)
plt.savefig('lpf.png')
plt.show()
#Application of hpf
f2 = hpf(f1[16:,:])
print(f2[-300:-200,1])
plt.plot(f2[:-100,0], f2[:-100,1])
plt.title('After Pan-Tompkins LPF+HPF')
plt.xlabel('time')
plt.ylabel('mV')
plt.grid(True)
plt.savefig('hpf.png')
plt.show()
raw data in CSV format:
timestamp,ecg_measurement
96813044,2.2336266040
96816964,2.1798632144
96820892,2.1505377292
96824812,2.1603128910
96828732,2.1554253101
96832660,2.1163244247
96836580,2.0576734542
96840500,2.0381231307
96844420,2.0527858734
96848340,2.0674486160
96852252,2.0283479690
96856152,1.9648094177
96860056,1.9208210945
96863976,1.9159335136
96867912,1.9208210945
96871828,1.8768328666
96875756,1.7986314296
96879680,1.7448680400
96883584,1.7155425548
96887508,1.7057673931
96891436,1.6520038604
96895348,1.5591397285
96899280,1.4809384346
96903196,1.4467253684
96907112,1.4369501113
96911032,1.3978494453
96914956,1.3440860509
96918860,1.2952101230
96922788,1.3000977039
96926684,1.3343108892
96930604,1.3440860509
96934516,1.3489736318
96938444,1.3294233083
96942364,1.3782991170
96946284,1.4222873687
96950200,1.4516129493
96954120,1.4369501113
96958036,1.4320625305
96961960,1.4565005302
96965872,1.4907135963
96969780,1.5053763389
96973696,1.4613881111
96977628,1.4125122070
96981548,1.4076246261
96985476,1.4467253684
96989408,1.4809384346
96993324,1.4760508537
96997236,1.4711632728
97001160,1.4907135963
97005084,1.5444769859
97008996,1.5982404708
97012908,1.5835777282
97016828,1.5591397285
97020756,1.5786901473
97024676,1.6324535369
97028604,1.6911046504
97032516,1.6959922313
97036444,1.6764417648
97040364,1.6813293457
97044296,1.7155425548
97048216,1.7448680400
97052120,1.7253177165
97056048,1.6911046504
97059968,1.6911046504
97063880,1.7302052974
97067796,1.7741935253
97071724,1.7693059444
97075644,1.7350928783
97079564,1.7595307826
97083480,1.8719452857
97087396,2.0381231307
97091316,2.2482893466
97095244,2.4828934669
97099156,2.7468230724
97103088,2.9960899353
97106996,3.0987291336
97110912,2.9178886413
97114836,2.5171065330
97118756,2.0185728073
97122668,1.5053763389
97126584,1.1094819307
97130492,0.8015640258
97134396,0.5767350673
97138308,0.4545454502
97142212,0.4349951267
97146124,0.4692081928
97150020,0.4887585639
97153924,0.4594330310
97157828,0.4105571746
97161740,0.3861192512
97165660,0.3763440847
97169580,0.3714565038
97173492,0.3225806236
97177404,0.2639296054
97181316,0.2394916772
97185236,0.2297165155
97189148,0.2443792819
97193060,0.2248289346
97196972,0.1857282543
97200900,0.1808406734
97204812,0.2199413537
97208732,0.2492668628
97212652,0.2443792819
97216572,0.2199413537
97220484,0.2248289346
97224404,0.2834799575
97228316,0.3274682044
97232228,0.3665689229
97236132,0.3861192512
97240036,0.4398827075
97243936,0.5083088874
97247836,0.6109481811
97251748,0.7086998939
97255660,0.7771260738
97259568,0.8553275108
97263476,0.9775171279
97267392,1.1094819307
97271308,1.1974585056
97275228,1.2512218952
97279148,1.2952101230
97283056,1.3734115362
97286992,1.4760508537
97290900,1.5493645668
97294820,1.5738025665
97298740,1.5982404708
97302652,1.6471162796
97306584,1.7106549739
97310500,1.7546432018
97314420,1.7546432018
97318340,1.7644183635
97322272,1.8084066390
97326168,1.8621701240
97330072,1.8963831901
97333988,1.8817204475
97337912,1.8572825431
97341840,1.8670577049
97345748,1.8866080284
97349668,1.8768328666
97353580,1.8230694770
97357500,1.7595307826
97361424,1.7302052974
97365332,1.7350928783
97369252,1.6959922313
97373168,1.6226783752
97377092,1.5298142433
97381012,1.4613881111
97384940,1.4320625305
97388860,1.4076246261
97392780,1.3440860509
97396676,1.2658846378
97400604,1.2121212482
97404532,1.1974585056
97408444,1.1779080629
97412356,1.1192570924
97416264,1.0361680984
97420164,0.9628542900
97424068,0.9286412239
97427988,0.9042033195
97431892,0.8406646728
97435804,0.7575757503
97439708,0.6940371513
97443628,0.6793744087
97447540,0.6793744087
97451452,0.6549364089
97455356,0.6060606002
97459240,0.5767350673
97463140,0.6011730194
97467044,0.6451612472
97470964,0.6842619895
97474884,0.6891495704
97478796,0.7184750556
97482700,0.8064516067
97486612,0.8846529960
97490516,0.9335288047
97494428,0.9530791282
97498340,0.9481915473
97502256,0.9726295471
97506156,0.9921798706
97510060,0.9726295471
97513980,0.8846529960
97517884,0.7869012832
97521796,0.7086998939
97525692,0.6549364089
97529604,0.5913978099
97533516,0.4887585639
97537428,0.3567937374
97541348,0.2639296054
97545260,0.2003910064
97549148,0.1417399787
97553060,0.0928641223
97556980,0.0537634420
97560892,0.0342130994
97564804,0.0146627559
97568708,0.0244379281
97572628,0.0048875851
97576500,0.0000000000
97580324,0.0000000000
97584172,0.0097751703
97588060,0.0244379281
97591980,0.0195503416
97595900,0.0146627559
97599812,0.0488758563
97603724,0.1319648027
97607628,0.2248289346
97611548,0.3030303001
97615444,0.3665689229
97619364,0.4496578693
97623276,0.5718474864
97627176,0.7038123130
97631076,0.8064516067
97634988,0.8699902534
97638900,0.9384163856
97642816,1.0361680984
97646720,1.1485825777
97650644,1.2365591526
97654572,1.2658846378
97658492,1.2805473804
97662404,1.3294233083
97666324,1.3782991170
97670244,1.3831867027
97674148,1.3489736318
97678068,1.3049852848
97681988,1.2903225421
97685908,1.3000977039
97689812,1.3098728656
97693728,1.2463343143
97697648,1.1876833438
97701568,1.1681329011
97705488,1.1876833438
97709412,1.1827956390
97713328,1.1339198350
97717244,1.0752688646
97721144,1.0557184219
97725056,1.0703812837
97728972,1.0850440216
97732872,1.0752688646
97736788,1.0459432601
97740696,1.0508308410
97744600,1.0948191833
97748520,1.1290322542
97752444,1.1192570924
97756364,1.0850440216
97760272,1.0801564407
97764168,1.1094819307
97768072,1.1339198350
97771996,1.1143695116
97775920,1.0557184219
97779840,1.0166177749
97783756,0.9970674514
97787668,0.9921798706
97791580,0.9530791282
97795500,0.8846529960
97799412,0.8504399299
97803316,0.8455523490
97807212,0.8699902534
97811124,0.8699902534
97815028,0.8308895111
97818940,0.8064516067
97822844,0.8211143493
97826756,0.8651026725
97830668,0.9042033195
97834572,0.8895405769
97838476,0.8993157386
97842396,0.9530791282
97846304,1.0410556793
97850204,1.0850440216
97854112,1.0899316024
97858020,1.1192570924
97861940,1.2267839908
97865860,1.4320625305
97869772,1.6911046504
97873688,1.9892473220
97877604,2.3020527362
97881524,2.6197457313
97885452,2.8299119949
97889372,2.7761485576
97893292,2.4535679817
97897216,1.9745845794
97901136,1.4956011772
97905052,1.0899316024
97908964,0.8260019302
97912864,0.6695992469
97916772,0.6353860855
97920684,0.7038123130
97924588,0.8553275108
97928496,1.0215053558
97932408,1.1388074159
97936324,1.2023460865
97940228,1.2463343143
97944148,1.3098728656
97948064,1.3734115362
97951988,1.3929618644
97955912,1.3734115362
97959836,1.3636363744
97963764,1.3782991170
97967684,1.4173997879
97971612,1.4173997879
97975540,1.3880742835
97979460,1.3831867027
97983372,1.4027370452
97987292,1.4467253684
97991216,1.4565005302
97995124,1.4320625305
97999036,1.4173997879
98002964,1.4662756919
98006884,1.5249266624
98010812,1.5689149856
98014740,1.5689149856
98018668,1.5689149856
98022592,1.6129032135
98026516,1.6715541839
98030436,1.6911046504
98034348,1.6617790222
98038268,1.6422286987
98042192,1.6715541839
98046124,1.7204301357
98050032,1.7399804592
98053952,1.7155425548
98057880,1.6911046504
98061800,1.7106549739
98065716,1.7595307826
98069636,1.7937438488
98073556,1.7888562679
98077476,1.7741935253
98081408,1.8084066390
98085312,1.8719452857
98089228,1.9305962562
98093144,1.9257086753
98097048,1.9257086753
98100968,1.9599218368
98104884,2.0332355499
98108804,2.0967741012
98112724,2.1016616821
98116652,2.0869989395
98120564,2.0967741012
98124484,2.1456501483
98128404,2.1847507953
98132324,2.1749756336
98136252,2.1212120056
98140156,2.0967741012
98144068,2.1114368438
98147996,2.0967741012
98151908,2.0430107116
98155824,1.9501466751
98159748,1.8817204475
98163664,1.8475073814
98167584,1.8377322196
98171508,1.7937438488
98175440,1.7253177165
98179364,1.7057673931
98183296,1.7106549739
98187200,1.7448680400
98191108,1.7546432018
98195032,1.7302052974
98198952,1.7302052974
98202868,1.7741935253
98206800,1.8426198005
98210712,1.8866080284
98214628,1.8914956092
98218548,1.8914956092
98222472,1.9403715133
98226396,2.0087976455
98230308,2.0527858734
98234212,2.0527858734
98238132,2.0527858734
98242044,2.0869989395
98245964,2.1407625675
98249892,2.1798632144
98253812,2.1749756336
98257740,2.1652004718
98261660,2.1896383762
98265588,2.2385141849
98269516,2.2678396701
98273444,2.2385141849
98277364,2.1896383762
98281292,2.1798632144
98285212,2.1994135379
98289140,2.2091886997
98293052,2.1798632144
98296980,2.1212120056
98300892,2.0918865203
98304804,2.1114368438
98308732,2.1163244247
98312660,2.0674486160
98316572,2.0087976455
98320480,1.9892473220
98324392,1.9892473220
98328308,2.0087976455
98332216,1.9892473220
98336132,1.9501466751
98340048,1.9354838371
98343972,1.9696969985
98347888,1.9843597412
98351812,1.9696969985
98355736,1.9159335136
98359664,1.8866080284
98363576,1.9012707710
98367484,1.9305962562
98371408,1.9208210945
98375324,1.8817204475
98379240,1.8719452857
98383156,1.8817204475
98387072,1.9305962562
98390984,1.9403715133
98394904,1.9159335136
98398832,1.9012707710
98402744,1.9354838371
98406672,1.9794721603
98410584,1.9941349029
98414492,1.9696969985
98418416,1.9550342559
98422336,1.9843597412
98426260,2.0430107116
98430164,2.0723361968
98434076,2.0527858734
98437988,2.0381231307
98441900,2.0625610351
98445820,2.1065492630
98449740,2.1309874057
98453660,2.1065492630
98457572,2.0869989395
98461492,2.0918865203
98465404,2.1456501483
98469324,2.1847507953
98473236,2.1749756336
98477148,2.1505377292
98481052,2.1652004718
98484972,2.1945259571
98488900,2.2287390232
98492820,2.2091886997
98496732,2.1700880527
98500644,2.1652004718
98504556,2.2091886997
98508476,2.2531769275
98512404,2.2336266040
98516324,2.1994135379
98520244,2.2043011188
98524152,2.2531769275
98528068,2.2873899936
98531988,2.2727272510
98535908,2.2238514423
98539836,2.1994135379
98543764,2.2336266040
98547676,2.2580645084
98551588,2.2482893466
98555508,2.1994135379
98559436,2.1652004718
98563356,2.1603128910
98567268,2.1700880527
98571164,2.1309874057
98575068,2.0527858734
98578992,1.9843597412
98582920,1.9648094177
98586840,1.9696969985
98590756,1.9501466751
98594680,1.8963831901
98598596,1.8523949623
98602528,1.8572825431
98606456,1.8621701240
98610376,1.8670577049
98614292,1.8328446388
98618204,1.8132943153
98622132,1.8426198005
98626048,1.8963831901
98629968,1.9257086753
98633892,1.8914956092
98637808,1.8670577049
98641716,1.8914956092
98645640,1.9941349029
98649556,2.1456501483
98653476,2.3313782215
98657404,2.5708699226
98661316,2.8885631561
98665236,3.2306940555
98669148,3.4799609184
98673064,3.4604105949
98676972,3.1769306659
98680884,2.7614858150
98684796,2.2678396701
98688720,1.8230694770
98692628,1.4418377876
98696556,1.2023460865
98700476,1.1241446733
98704400,1.1876833438
98708316,1.3098728656
98712228,1.4125122070
98716148,1.4858260154
98720060,1.5493645668
98723988,1.6275659561
98727908,1.6764417648
98731836,1.6911046504
98735748,1.6617790222
98739676,1.6471162796
98743608,1.6715541839
98747532,1.7057673931
98751452,1.7057673931
98755372,1.6568914413
98759300,1.6275659561
98763220,1.6422286987
98767140,1.6862169265
98771056,1.6911046504
98774964,1.6617790222
98778884,1.6568914413
98782812,1.6911046504
98786728,1.7448680400
98790632,1.7790811061
98794544,1.7693059444
98798452,1.7644183635
98802384,1.7888562679
98806312,1.8279570579
98810224,1.8426198005
98814132,1.8181818962
98818044,1.7986314296
98821972,1.8181818962
98825900,1.8523949623
98829832,1.8719452857
98833760,1.8377322196
98837684,1.8035190582
98841596,1.7986314296
98845528,1.8377322196
98849456,1.8670577049
98853368,1.8523949623
98857292,1.8181818962
98861220,1.8328446388
98865140,1.8866080284
98869048,1.9305962562
98872968,1.9305962562
98876888,1.9012707710
98880800,1.9208210945
98884704,1.9599218368
98888624,1.9892473220
98892544,1.9599218368
98896464,1.8866080284
98900376,1.8426198005
98904296,1.8377322196
98908216,1.8328446388
98912132,1.7839686870
98916040,1.7008798122
98919956,1.6471162796
98923884,1.6373411178
98927812,1.6324535369
98931740,1.5982404708
98935644,1.5151515007
98939564,1.4613881111
98943492,1.4418377876
98947424,1.4271749496
98951348,1.3685239553
98955260,1.2707722187
98959180,1.1925709247
98963092,1.1534701585
98967008,1.1339198350
98970932,1.1045943498
98974840,1.0215053558
98978748,0.9677418708
98982660,0.9579667091
98986572,0.9775171279
98990492,0.9824047088
98994396,0.9237536430
98998308,0.8748778343
99002212,0.8797654151
99006132,0.9188660621
99010036,0.9286412239
99013956,0.9090909004
99017836,0.8895405769
99021740,0.9042033195
99025644,0.9530791282
99029556,0.9921798706
99033468,0.9970674514
99037380,0.9872922897
99041296,1.0166177749
99045196,1.0752688646
99049100,1.1192570924
99053016,1.1290322542
99056940,1.0997067642
99060840,1.1094819307
99064744,1.1485825777
99068668,1.1925709247
99072588,1.2023460865
99076508,1.1925709247
99080428,1.2023460865
99084340,1.2658846378
99088252,1.3343108892
99092168,1.3587487936
99096084,1.3343108892
99100004,1.3294233083
99103924,1.3636363744
99107860,1.4027370452
99111772,1.3831867027
99115700,1.3343108892
99119628,1.3147605657
99123556,1.3343108892
99127480,1.3587487936
99131404,1.3538612127
99135324,1.3049852848
99139236,1.2756597995
99143156,1.2903225421
99147076,1.3196481466
99151012,1.3147605657
99154924,1.2707722187
99158844,1.2072336673
99162764,1.2023460865
99166676,1.2267839908
99170588,1.2365591526
99174508,1.1974585056
99178420,1.1632453203
99182340,1.1534701585
99186248,1.1876833438
99190164,1.1974585056
99194072,1.1583577394
99197996,1.1192570924
99201916,1.1192570924
99205832,1.1730204820
99209748,1.2072336673
99213668,1.2023460865
99217588,1.1779080629
99221488,1.1876833438
99225412,1.2267839908
99229332,1.2707722187
99233244,1.2609970569
99237152,1.2365591526
99241068,1.2463343143
99244988,1.2805473804
99248900,1.2952101230
99252820,1.2805473804
99256732,1.2316715717
99260660,1.2316715717
99264588,1.2854349613
99268512,1.3391984701
99272436,1.3538612127
99276364,1.3343108892
99280292,1.3391984701
99284212,1.3782991170
99288116,1.4271749496
99292040,1.4369501113
99295964,1.4076246261
99299892,1.4076246261
99303816,1.4662756919
99307740,1.5395894050
99311652,1.5640274047
99315564,1.5444769859
99319484,1.5444769859
99323412,1.5786901473
99327332,1.6275659561
99331252,1.6520038604
99335156,1.6422286987
99339076,1.6275659561
99343004,1.6422286987
99346924,1.6666666030
99350844,1.6568914413
99354764,1.6031280517
99358676,1.5542521476
99362604,1.5542521476
99366532,1.5835777282
99370460,1.5982404708
99374372,1.5835777282
99378300,1.5640274047
99382204,1.5835777282
99386132,1.6373411178
99390056,1.6715541839
99393980,1.6520038604
99397892,1.6275659561
99401812,1.6422286987
99405736,1.6862169265
99409664,1.7106549739
99413580,1.6911046504
99417500,1.6568914413
99421432,1.6715541839
99425348,1.7204301357
99429256,1.8084066390
99433164,1.9208210945
99437068,2.0918865203
99440980,2.3655912876
99444912,2.7321603298
99448828,3.0596284866
99452752,3.2453567981
99456680,3.1867058277
99460600,2.9374389648
99464516,2.5610947608
99468428,2.1163244247
99472356,1.6813293457
99476284,1.3343108892
99480200,1.1436949968
99484112,1.1339198350
99488036,1.2365591526
99491956,1.3440860509
99495864,1.4320625305
99499780,1.5298142433
99503708,1.6422286987
99507636,1.7350928783
99511556,1.7644183635
99515480,1.7399804592
99519396,1.7350928783
99523320,1.7448680400
99527220,1.7350928783
99531140,1.6862169265
99535064,1.5933528900
99538980,1.5102639198
99542892,1.4711632728
99546820,1.4467253684
99550748,1.3978494453
99554668,1.3049852848
99558588,1.2072336673
99562504,1.1485825777
99566428,1.1192570924
99570348,1.0752688646
99574256,1.0068426132
99578176,0.9384163856
99582084,0.9188660621
99585988,0.9188660621
99589900,0.9188660621
99593812,0.8895405769
99597716,0.8748778343
99601636,0.8651026725
99605552,0.9090909004
99609436,0.9481915473
99613356,0.9530791282
99617268,0.9237536430
99621180,0.9335288047
99625080,1.0019550323
99628980,1.0752688646
99632888,1.0801564407
99636792,1.0703812837
99640704,1.0899316024
99644616,1.1436949968
99648536,1.2170088291
99652444,1.2170088291
99656356,1.2023460865
99660268,1.2072336673
99664180,1.2561094760
99668084,1.3000977039
99671980,1.3147605657
99675900,1.2952101230
99679820,1.3000977039
99683728,1.3587487936
99687652,1.4027370452
99691568,1.4222873687
99695484,1.3978494453
99699404,1.3880742835
99703328,1.4173997879
99707248,1.4565005302
99711156,1.4760508537
99715064,1.4271749496
99718988,1.3929618644
99722908,1.3929618644
99726828,1.4076246261
99730748,1.3831867027
99734668,1.3147605657
99738580,1.2561094760
99742492,1.2414467334
99746420,1.2658846378
99750340,1.2658846378
99754252,1.2365591526
99758168,1.2121212482
99762084,1.2365591526
99766012,1.3000977039
99769916,1.3538612127
99773856,1.3685239553
99777780,1.3929618644
99781704,1.4662756919
99785620,1.5640274047
99789532,1.6568914413
99793460,1.6959922313
99797392,1.7057673931
99801312,1.7399804592
99805228,1.7937438488
99809148,1.8377322196
99813072,1.8377322196
99816996,1.8230694770
99820920,1.8475073814
99824840,1.9061583518
99828756,1.9501466751
99832680,1.9599218368
99836608,1.9501466751
99840536,1.9599218368
99844452,2.0087976455
99848364,2.0527858734
99852268,2.0527858734
99856184,2.0283479690
99860092,2.0185728073
99864012,2.0576734542
99867932,2.0967741012
99871836,2.0869989395
99875740,2.0478982925
99879652,2.0234603881
99883564,2.0527858734
99887484,2.1065492630
99891404,2.1163244247
99895332,2.0772237777
99899236,2.0527858734
99903156,2.0821113586
99907076,2.1065492630
99910996,2.1016616821
99914916,2.0576734542
99918828,2.0283479690
99922740,2.0430107116
99926652,2.0821113586
99930572,2.1016616821
99934492,2.0576734542
99938404,2.0332355499
99942316,2.0674486160
99946220,2.1309874057
99950124,2.1749756336
99954052,2.1652004718
99957972,2.1260998249
99961892,2.1456501483
99965804,2.1945259571
99969732,2.2336266040
99973644,2.2189638614
99977564,2.1945259571
99981492,2.2043011188
99985404,2.2482893466
99989332,2.2922775745
99993252,2.2580645084
99997164,2.2238514423
100001084,2.2189638614
100005044,2.2678396701
100009004,2.2776148319
100012956,2.2385141849
100016924,2.1798632144
100020892,2.1603128910
100024844,2.1798632144
100028804,2.2140762805
100032756,2.1798632144
100036716,2.1358749866
100040676,2.1163244247
100044644,2.1358749866
100048604,2.1603128910
100052556,2.1407625675
100056516,2.0967741012
100060468,2.0918865203
100064420,2.1163244247
100068384,2.1407625675
100072340,2.1065492630
100076292,2.0478982925
100080244,2.0332355499
100084196,2.0478982925
100088156,2.0674486160
100092100,2.0332355499
100096056,1.9696969985
100100004,1.9110459327
100103968,1.9061583518
100107928,1.9208210945
100111884,1.8768328666
100115844,1.8181818962
100119812,1.7888562679
100123776,1.8084066390
100127712,1.8475073814
100131672,1.8523949623
100135636,1.8181818962
100139604,1.8035190582
100143560,1.8377322196
100147524,1.8768328666
100151488,1.8719452857
100155448,1.8523949623
100159404,1.8132943153
100163376,1.8426198005
100167328,1.8963831901
100171276,1.9110459327
100175232,1.9061583518
100179188,1.9501466751
100183132,2.1016616821
100187084,2.3216030597
100191036,2.5904202461
100194996,2.8787879943
100198956,3.1769306659
100202916,3.4555230140
100206876,3.5826001167
100210836,3.4115347862
100214804,3.0205278396
100218748,2.5317692756
100222708,2.1016616821
100226676,1.7937438488
100230640,1.5933528900
100234592,1.4858260154
100238548,1.5053763389
100242500,1.6422286987
100246456,1.8377322196
100250424,1.9990224838
100254380,2.0967741012
100258340,2.1652004718
100262284,2.2434017658
100266236,2.3411533832
100270196,2.4242424964
100274140,2.4731183052
100278112,2.4975562095
100282068,2.5562071800
100286020,2.6441838741
100289992,2.7028348445
100293948,2.7077224254
100297908,2.7126100063
100301868,2.7468230724
100305820,2.8054740905
100309764,2.8250244140
100313724,2.7908113002
100317676,2.7370479106
100321636,2.7223851680
100325600,2.7419354915
100329556,2.7517106533
100333516,2.7126100063
100337468,2.6735093593
100341428,2.6686217784
100345392,2.7028348445
100349348,2.7272727489
100353316,2.6881721019
100357260,2.6441838741
100361212,2.6588466167
100365180,2.6832845211
100369140,2.7077224254
100373100,2.6783969402
100377052,2.6148581504
100381012,2.6001954078
100384960,2.6246333122
100388916,2.6490714550
100392860,2.6197457313
100396828,2.5659823417
100400788,2.5562071800
100404740,2.5806450843
100408692,2.6099705696
100412644,2.5904202461
100416588,2.5366568565
100420548,2.5268816947
100424508,2.5610947608
100428460,2.5953078269
100432412,2.5757575035
100436372,2.5171065330
100440324,2.4926686286
100444276,2.5219941139
100448228,2.5366568565
100452196,2.5073313713
100456148,2.4389052391
100460108,2.3949170112
100464068,2.3753666877
100468028,2.3655912876
100471988,2.3069403171
The trick seems to be the initial conditions. Load the first 13 values of input and output of low pass filter to zero and the bias goes away.
#low-pass filter
def lpf(x):
y = x.copy()
for n in range(13):
y[n,1] = 0
x[n,1] = 0
for n in range(len(x)):
if(n < 12):
continue
y[n,1] = 2*y[n-1,1] - y[n-2,1] + x[n,1] - 2*x[n-6,1] + x[n-12,1]
return y

Obtaining hyperpolarization depth from electrophysiological graph

I am working on electrophysiological data which is in .abf format.
I want to obtain the hyperpolarization depth as indicated above in the figure. This is what I have done so far;
import matplotlib.pyplot as plt
import pyabf
import pandas as pd
abf = pyabf.ABF("test.abf")
abf.setSweep(10) # I can access a given sweep. Here sweep 10
df = pd.DataFrame({'time': abf.sweepX, 'current':abf.sweepY})
df1 = df.loc[15650:15800]
df1.plot(x='time', y='current')
I am thinking to apply change in derivative to find the first point of interest (x1,y1) and then lower point (x2,y2), but it looks complex. I would appreciate if someone give some hint or procedure.
The dataset as follow,
time current
0.7825 -63.323975
0.78255 -63.171387
0.7826 -62.89673
0.78265 -62.713623
0.7827 -62.469482
0.78275 -62.37793
0.7828 -62.10327
0.78285 -61.950684
0.7829 -61.76758
0.78295 -61.584473
0.783 -61.401367
0.78305 -61.24878
0.7831 -61.035156
0.78315 -60.85205
0.7832 -60.72998
0.78325 -60.516357
0.7833 -60.455322
0.78335 -60.2417
0.7834 -60.08911
0.78345 -59.96704
0.7835 -59.814453
0.78355 -59.661865
0.7836 -59.509277
0.78365 -59.417725
0.7837 -59.23462
0.78375 -59.11255
0.7838 -58.95996
0.78385 -58.86841
0.7839 -58.685303
0.78395 -58.59375
0.784 -58.441162
0.78405 -58.34961
0.7841 -58.19702
0.78415 -58.044434
0.7842 -57.922363
0.78425 -57.769775
0.7843 -57.678223
0.78435 -57.434082
0.7844 -57.34253
0.78445 -56.9458
0.7845 -56.274414
0.78455 -54.96216
0.7846 -53.253174
0.78465 -51.208496
0.7847 -48.950195
0.78475 -46.325684
0.7848 -43.09082
0.78485 -38.42163
0.7849 -31.036377
0.78495 -22.033691
0.785 -13.397217
0.78505 -6.072998
0.7851 -0.61035156
0.78515 2.7160645
0.7852 3.9367676
0.78525 3.4179688
0.7853 1.3427734
0.78535 -1.4953613
0.7854 -5.0964355
0.78545 -9.185791
0.7855 -13.641357
0.78555 -18.249512
0.7856 -23.132324
0.78565 -27.98462
0.7857 -32.714844
0.78575 -37.261963
0.7858 -41.47339
0.78585 -45.22705
0.7859 -48.553467
0.78595 -51.54419
0.786 -53.985596
0.78605 -56.18286
0.7861 -58.013916
0.78615 -59.539795
0.7862 -60.760498
0.78625 -61.88965
0.7863 -62.652588
0.78635 -63.323975
0.7864 -63.934326
0.78645 -64.2395
0.7865 -64.60571
0.78655 -64.78882
0.7866 -65.00244
0.78665 -64.971924
0.7867 -65.093994
0.78675 -65.03296
0.7868 -64.971924
0.78685 -64.819336
0.7869 -64.78882
0.78695 -64.66675
0.787 -64.48364
0.78705 -64.42261
0.7871 -64.2395
0.78715 -64.11743
0.7872 -63.964844
0.78725 -63.842773
0.7873 -63.659668
0.78735 -63.568115
0.7874 -63.446045
0.78745 -63.26294
0.7875 -63.171387
0.78755 -62.98828
0.7876 -62.89673
0.78765 -62.74414
0.7877 -62.713623
0.78775 -62.530518
0.7878 -62.438965
0.78785 -62.37793
0.7879 -62.25586
0.78795 -62.164307
0.788 -62.042236
0.78805 -62.01172
0.7881 -61.88965
0.78815 -61.88965
0.7882 -61.73706
0.78825 -61.706543
0.7883 -61.645508
0.78835 -61.61499
0.7884 -61.523438
0.78845 -61.462402
0.7885 -61.431885
0.78855 -61.340332
0.7886 -61.37085
0.78865 -61.279297
0.7887 -61.279297
0.78875 -61.157227
0.7888 -61.187744
0.78885 -61.09619
0.7889 -61.157227
0.78895 -61.12671
0.789 -61.09619
0.78905 -61.12671
0.7891 -61.00464
0.78915 -61.00464
0.7892 -60.97412
0.78925 -60.97412
0.7893 -60.943604
0.78935 -61.00464
0.7894 -60.913086
0.78945 -60.97412
0.7895 -60.943604
0.78955 -60.913086
0.7896 -60.943604
0.78965 -60.85205
0.7897 -60.85205
0.78975 -60.821533
0.7898 -60.88257
0.78985 -60.88257
0.7899 -60.913086
0.78995 -60.88257
0.79 -60.913086
We can plot the difference in current between consecutive points (which essentially is to a constant factor the derivative, since times are evenly spaced). First chart shows the actual diffs. Based on this we can set some threshold, such as 0.3, and apply it to filter the main DataFrame. The filtered values are shown in orange on the second chart:
fig, ax = plt.subplots(2, figsize=(8,8))
# plot derivative
df['current'].diff().plot(ax=ax[0])
# current
threshold = 0.4
df['filtered'] = df.loc[df['current'].diff().abs() > threshold]
df.plot(ax=ax[1])
# add spans
x = df['filtered'].dropna()
ax[1].axhspan(x.iloc[0], x.iloc[-1], alpha=0.3, edgecolor='skyblue', facecolor="none", hatch='////')
ax[1].axvspan(x.index.min(), x.index.max(), alpha=0.3, edgecolor='orange', facecolor="none", hatch='\\\\')
Output:
If you're interested in range values, you can dropna values in the filtered subset and find min and max from the index:
print('min', df['filtered'].dropna().index.min())
print('max', df['filtered'].dropna().index.max())
Output:
min 0.78445
max 0.7865
For the value of the gap you can use:
abs(df['filtered'].dropna().iloc[-1] - df['filtered'].dropna().iloc[0])
Output:
7.6599100000000035
Note: We can alternatively also get left edges of these spans as points where diff in the point is lower than the threshold and diff in the next point is higher than the threshold, and similarly for the right edges. This would also work in case we have multiple peaks:
threshold = 0.3
x = df['current'].diff().abs()
spanA = df.loc[(x < threshold) & (x.shift(-1) >= threshold)]
spanB = df.loc[(x >= threshold) & (x.shift(-1) < threshold)]
print(spanA)
current
time
0.7844 -57.34253
print(spanB)
current
time
0.7865 -64.60571

Overwrite GPS coordinates in Image Exif using Python 3.6

I am trying to transform image geotags so that images and ground control points lie in the same coordinate system inside my software (Pix4D mapper).
The answer here says:
Exif data is standardized, and GPS data must be encoded using
geographical coordinates (minutes, seconds, etc) described above
instead of a fraction. Unless it's encoded in that format in the exif
tag, it won't stick.
Here is my code:
import os, piexif, pyproj
from PIL import Image
img = Image.open(os.path.join(dirPath,fn))
exif_dict = piexif.load(img.info['exif'])
breite = exif_dict['GPS'][piexif.GPSIFD.GPSLatitude]
lange = exif_dict['GPS'][piexif.GPSIFD.GPSLongitude]
breite = breite[0][0] / breite[0][1] + breite[1][0] / (breite[1][1] * 60) + breite[2][0] / (breite[2][1] * 3600)
lange = lange[0][0] / lange[0][1] + lange[1][0] / (lange[1][1] * 60) + lange[2][0] / (lange[2][1] * 3600)
print(breite) #48.81368778730952
print(lange) #9.954511162420633
x, y = pyproj.transform(wgs84, gk3, lange, breite) #from WGS84 to GaussKrüger zone 3
print(x) #3570178.732528623
print(y) #5408908.20172699
exif_dict['GPS'][piexif.GPSIFD.GPSLatitude] = [ ( (int)(round(y,6) * 1000000), 1000000 ), (0, 1), (0, 1) ]
exif_bytes = piexif.dump(exif_dict) #error here
img.save(os.path.join(outPath,fn), "jpeg", exif=exif_bytes)
I am getting struct.error: argument out of range in the dump method. The original GPSInfo tag looks like: {0: b'\x02\x03\x00\x00', 1: 'N', 2: ((48, 1), (48, 1), (3449322402, 70000000)), 3: 'E', 4: ((9, 1), (57, 1), (1136812930, 70000000)), 5: b'\x00', 6: (3659, 10)}
I am guessing I have to offset the values and encode them properly before writing, but have no idea what is to be done.
It looks like you are already using PIL and Python 3.x, not sure if you want to continue using piexif but either way, you may find it easier to convert the degrees, minutes, and seconds into decimal first. It looks like you are trying to do that already but putting it in a separate function may be clearer and account for direction reference.
Here's an example:
def get_decimal_from_dms(dms, ref):
degrees = dms[0][0] / dms[0][1]
minutes = dms[1][0] / dms[1][1] / 60.0
seconds = dms[2][0] / dms[2][1] / 3600.0
if ref in ['S', 'W']:
degrees = -degrees
minutes = -minutes
seconds = -seconds
return round(degrees + minutes + seconds, 5)
def get_coordinates(geotags):
lat = get_decimal_from_dms(geotags['GPSLatitude'], geotags['GPSLatitudeRef'])
lon = get_decimal_from_dms(geotags['GPSLongitude'], geotags['GPSLongitudeRef'])
return (lat,lon)
The geotags in this example is a dictionary with the GPSTAGS as keys instead of the numeric codes for readability. You can find more detail and the complete example from this blog post: Getting Started with Geocoding Exif Image Metadata in Python 3
After much hemming & hawing I reached the pages of py3exiv2 image metadata manipulation library. One will find exhaustive lists of the metadata tags as one reads through but here is the list of EXIF tags just to save few clicks.
It runs smoothly on Linux and provides many opportunities to edit image-headers. The documentation is also quite clear. I recommend this as a solution and am interested to know if it solves everyone else's problems as well.

How to communicate through a serial port with python?

I am creating a Solar Array Simulator with python to simulate the voltage and current being created with different angles and intensities of sunlight as a satellite orbits around the earth. I have a very simple program that outputs the voltage and current just with the angle (no orbiting parameters yet). However, I need it to communicate the outputs generated with a E4350B model solar array simulator through a serial port, and I don't know where to start. I have installed pip and used that to install PySerial but do not know what to do from there. How do I communicate the voltage and amp outputs to the simulator through COM ports? Here is what I have for my program that runs the simulator.
from math import sin,radians,pi
import time
'''Needed information:
Voc (per one cell) = 2,680mV
Vmp (per one cell = 2,325mV
Isc (per one cell) = 453mA
Imp (per one cell) = 434mA
angular velocity = .05d/s'''
#Timer eclipse set for total eclipse tim ein seconds
timeEclipse = 1800
#total eclipse time = 30 mins
#Timer sun set for how many seconds it takes to change one degree
timeSun = 20
#Total sun exposer time = 60 mins
#Find the Vmp, Voc, Imp, and Isc from Beta angles 0 - 180
def Exposure():
tSun = timeSun
for x in range(0,181):
angle = sin(radians(x))
Voc = 2680 * angle
Vmp = 2325 * angle
#Amps are going to be a function of voltage
Isc = 453 * angle
Imp = 434 * angle
#
print('At angle ',x,' Vmp = ',Vmp,
'mV, Voc = ',Voc,'mV, Isc = ',Isc,'mA, and Imp = ',Imp,'mA')
time.sleep(tSun)
#Simulate time during eclipse
#Outputs nothing
def Eclipse():
tEclipse = timeEclipse
time.sleep(tEclipse)
#Run loop through Exposure and eclipse
def Run():
run = True
while(run):
Exposure()
Eclipse()
P.S. For anybody who dabbles in a little bit of physics, I need a way to find the current as a function of the voltage at every angle.

Resources