I have an Android plugin for Unity3D that decodes h264 video into a surface. This works on my S6 with Android 5.1.1. However, on my S9 running Android 8, output buffers are never dequeued.
My code for creating the decoder is straightforward:
codec = AMediaCodec_createDecoderByType ("video/avc");
AMediaFormat * format = AMediaFormat_new ();
AMediaFormat_setString (format, AMEDIAFORMAT_KEY_MIME, "video/avc");
AMediaFormat_setInt32 (format, AMEDIAFORMAT_KEY_WIDTH, 4096);
AMediaFormat_setInt32 (format, AMEDIAFORMAT_KEY_HEIGHT, 1536);
AMediaFormat_setInt32 (format, AMEDIAFORMAT_KEY_FRAME_RATE, 30);
AMediaCodec_configure (codec, format, window, NULL, 0);
AMediaCodec_start (codec);
AMediaFormat_delete (format);
Similarly, my code for queueing into the codec is relatively standard
ssize_t buffIdx = AMediaCodec_dequeueInputBuffer (codec, DequeueTimeoutUS);
if (buffIdx < 0)
continue;
uint8_t* inputBuff = AMediaCodec_getInputBuffer (codec, buffIdx, &buffsize);
// I would copy into the input buffer here...
AMediaCodec_queueInputBuffer (codec, buffIdx, 0, copySize, pts, flags);
And my code for pulling from the decoder:
AMediaCodecBufferInfo info;
ssize_t status = AMediaCodec_dequeueOutputBuffer (codec, &info, DequeueOutputTimeout);
if (status == AMEDIACODEC_INFO_TRY_AGAIN_LATER)
continue;
// I would update the surface texture here...
AMediaCodec_releaseOutputBuffer (codec, status, info.size != 0);
Unfortunately, dequeueing only returns AMEDIACODEC_INFO_TRY_AGAIN_LATER. Here are the logs from it running on the S9+
02-14 15:00:50.346 8005 8158 I ACodec : [] Now uninitialized
02-14 15:00:50.347 8005 8160 I ACodec : [] onAllocateComponent
02-14 15:00:50.353 8005 8160 I OMXClient: Treble IOmx obtained
02-14 15:00:50.353 8005 8160 I ACodec : Set Google AAC Dec for aacProfile 0
02-14 15:00:50.357 2692 7767 I OMXMaster: makeComponentInstance(OMX.qcom.video.decoder.avc) in omx#1.0-service process
02-14 15:00:50.359 684 684 I android.hardware.wifi#1.0-service: getLinkLayerStats
02-14 15:00:50.365 2692 7767 E : Service not available yet
02-14 15:00:50.388 2692 7767 I OMX-VDEC-1080P: Video slvp perflock acquired
02-14 15:00:50.389 2692 7767 I OMX-VDEC-1080P: component_init: OMX.qcom.video.decoder.avc : fd=22
02-14 15:00:50.389 2692 7767 E OMX-VDEC-1080P: Unsupported output color format for c2d (2141391876)
02-14 15:00:50.389 2692 7767 E OMX-VDEC-1080P: Setting color format failed
02-14 15:00:50.396 2692 7767 I OMX-VDEC-1080P: omx_vdec::component_init() success : fd=22
02-14 15:00:50.397 8005 8160 I ACodec : [OMX.qcom.video.decoder.avc] Now Loaded
02-14 15:00:51.818 8005 8159 D SurfaceUtils: connecting to surface 0xc4b86808, reason connectToSurface
02-14 15:00:51.818 8005 8159 I MediaCodec: [OMX.qcom.video.decoder.avc] setting surface generation to 8197121
02-14 15:00:51.818 8005 8159 D SurfaceUtils: disconnecting from surface 0xc4b86808, reason connectToSurface(reconnect)
02-14 15:00:51.818 8005 8159 D SurfaceUtils: connecting to surface 0xc4b86808, reason connectToSurface(reconnect)
02-14 15:00:51.819 2687 32660 I ResourceManagerService: ResourceManagerService checkCodecCapacity pid 8005 clientId 3218327712 useExtended 0
02-14 15:00:51.820 8005 8160 I ACodec : [HW_HDR] HDR-OFF 0
02-14 15:00:51.820 8005 8160 W DirectStreamingProxy: app-pid not found. use getpid(). pid = 8005
02-14 15:00:51.821 8005 8160 D DirectStreamingProxy: pid = 8005
02-14 15:00:51.823 2692 7767 E OMX-VDEC-1080P: Enable/Disable allocate-native-handle allowed only on input port!
02-14 15:00:51.823 2692 7767 E OMX-VDEC-1080P: set_parameter: Error: 0x80001019, setting param 0x7f00005d
02-14 15:00:51.823 2692 7767 E OMXNodeInstance: setParameter(0xee709444:qcom.decoder.avc, OMX.google.android.index.allocateNativeHandle(0x7f00005d): Output:1 en=0) ERROR: UnsupportedSetting(0x80001019)
02-14 15:00:51.823 2692 7767 E OMX-VDEC-1080P: Extension: OMX.google.android.index.storeANWBufferInMetadata not implemented
02-14 15:00:51.824 8005 8160 I ACodec : [OMX.qcom.video.decoder.avc] OMX_QCOM_FramePacking_OnlyOneCompleteFrame is setting
02-14 15:00:51.824 2676 3409 V APM_AudioPolicyManager: getOutputForDevice() returns output 29
02-14 15:00:51.824 2692 7767 E OMX-VDEC-1080P: Extension: OMX.google.android.index.storeANWBufferInMetadata not implemented
02-14 15:00:51.824 2692 7767 E OMX-VDEC-1080P: Extension: OMX.google.android.index.configureVideoTunnelMode not implemented
02-14 15:00:51.824 2692 7767 E OMX-VDEC-1080P: Extension: OMX.google.android.index.useAndroidNativeBuffer is supported
02-14 15:00:51.824 2692 7767 E OMX-VDEC-1080P: Setparameter: unknown param 2130706434
02-14 15:00:51.824 2692 7767 E OMX-VDEC-1080P: set_parameter: Error: 0x8000101a, setting param 0x7f000002
02-14 15:00:51.824 2692 7767 E OMXNodeInstance: setParameter(0xee709444:qcom.decoder.avc, ??(0x7f000002)) ERROR: UnsupportedIndex(0x8000101a)
02-14 15:00:51.824 8005 8160 W ACodec : Fail to set FramePackingMode(-1010)
02-14 15:00:51.824 2676 3409 I AudioFlinger: add dynamic flag, can move to deep thread, session 601
02-14 15:00:51.824 8005 8160 I ExtendedACodec: setupVideoDecoder()
02-14 15:00:51.828 8005 8160 I ExtendedACodec: Decoder will be in frame by frame mode
02-14 15:00:51.828 2692 2845 E OMX-VDEC-1080P: Setparameter: unknown param 2130706434
02-14 15:00:51.828 2692 2845 E OMX-VDEC-1080P: set_parameter: Error: 0x8000101a, setting param 0x7f000002
02-14 15:00:51.828 2692 2845 E OMXNodeInstance: setParameter(0xee709444:qcom.decoder.avc, ??(0x7f000002)) ERROR: UnsupportedIndex(0x8000101a)
02-14 15:00:51.828 8005 8160 W ExtendedACodec: Failed to set frame packing format on component
02-14 15:00:51.828 8005 8160 I SmartFittingClass: Create SmartFitting Version 2.0
02-14 15:00:51.830 2689 2887 I RemoteDisplayService: There is no RemoteDisplay
02-14 15:00:51.831 8005 8160 I SmartFittingClass: Init, [State:UNINITIALIZED] pid: 8005
02-14 15:00:51.832 3051 3365 D CodecSolution: setSmartFittingPid : 8005
02-14 15:00:51.833 3051 3365 D CodecSolution: setSmartFittingMode : 0
02-14 15:00:51.833 3051 3434 D CodecSolution: handleMessage : 202
02-14 15:00:51.836 2692 7768 E OMX-VDEC-1080P: Does not handle dataspace request
02-14 15:00:51.836 2692 7768 E OMXNodeInstance: getConfig(0xee709444:qcom.decoder.avc, ??(0x7f000062)) ERROR: UnsupportedSetting(0x80001019)
02-14 15:00:51.845 8005 8160 I ACodec : [OMX.qcom.video.decoder.avc] Now Loaded->Idle
02-14 15:00:51.858 2692 2900 E OMX-VDEC-1080P: Extension: OMX.google.android.index.AndroidNativeBufferConsumerUsage not implemented
02-14 15:00:51.858 8005 8160 D SurfaceUtils: set up nativeWindow 0xc4b86808 for 4096x1536, color 0x7fa30c06, rotation 0, usage 0x20402900
02-14 15:00:51.859 8005 8160 I ACodec : [OMX.qcom.video.decoder.avc] configureOutputBuffersFromNativeWindow setBufferCount :17, minUndequeuedBuffers : 5
02-14 15:00:51.860 3051 3365 W CodecSolution: PackageInfo is null.
02-14 15:00:51.860 8005 8175 I SmartFittingClass: InitialCheck, WhiteListStatus returned from CodecSolution : 0
02-14 15:00:51.860 8005 8175 I SmartFittingClass: InitialCheck,
02-14 15:00:51.864 3051 3365 D CodecSolution: reportMediaStatisticsEvent: action=1#1986621044;category=2#1986621044;label=1936222260#1986621044
02-14 15:00:51.864 3051 3365 D MediaStatisticsEvent: MediaStatisticsEvent: action=1#1986621044;category=2#1986621044;label=1936222260#1986621044
02-14 15:00:51.864 3051 3365 D MediaStatisticsEvent: Category.valueOf: 2
02-14 15:00:51.864 3051 3365 D MediaStatisticsEvent: Action.valueOf: 1
02-14 15:00:51.864 3051 3365 D MediaStatisticsEvent: Label.valueOf: 0x73686834
02-14 15:00:51.864 3051 3365 D MediaStatisticsEvent: category: VDEC
02-14 15:00:51.864 3051 3365 D MediaStatisticsEvent: action: INSTANTIATE
02-14 15:00:51.864 3051 3365 D MediaStatisticsEvent: label: SEC_HW_H264
02-14 15:00:51.865 3051 3434 D CodecSolution: handleMessage : 900
02-14 15:00:51.865 3051 3434 D CodecSolution: event : VDEC
02-14 15:00:51.865 3051 3434 D Logging : appId: com.samsung.android.codecsolution, feature: VDEC, extra: SEC_HW_H264, value: -1
02-14 15:00:51.865 2687 1137 I ResourceManagerService: addMediaInfo -(PID : 8005, clientID : 3218327712)
02-14 15:00:51.865 2687 1137 I ResourceManagerService: MediaInfo add 6291456 (width 4096 height 1536) remained 11403264
02-14 15:00:51.865 2687 1137 I ResourceManagerService: getMediaResourceInfo resourceType : 2, size : 1
02-14 15:00:51.865 2687 1137 I ResourceManagerService: getMediaResourceInfo (PID : 8005, clientID : 3218327712, non-secure-codec/video-codec:1, 4096x1536(fps:30) - SoftCodec : No, Encorder : No)
02-14 15:00:51.865 3051 3365 V ResourceManagerHelper-JNI: JNIMediaResourceHelper::notify eventType : 1, ext1 : 0, ext2 : 0
02-14 15:00:51.865 3051 3365 V ResourceManagerHelper-JNI: notify eventType : 1, ext1 : 0, ext2 : 0
02-14 15:00:51.865 8005 8160 I ACodec : [OMX.qcom.video.decoder.avc] Now Idle->Executing
02-14 15:00:51.866 3051 3365 I SemMediaResourceHelper: makeMediaResourceInfo mOwnResourceEventExcluded : false, mPid : 3051
02-14 15:00:51.866 3051 3365 I SemMediaResourceHelper: [1] makeMediaResourceInfo resourceType : 2 isSecured : false, pid : 8005, client id : 3218327712
02-14 15:00:51.866 8005 8160 I ACodec : [OMX.qcom.video.decoder.avc] Now Executing
02-14 15:00:51.870 8005 8160 W GrallocMapperPassthrough: buffer descriptor with invalid usage bits 0x2000
The last line about invalid usage bits repeats roughly in-step with me queueing the input buffers
Turns out this was because I was incidentally flagging every buffer with BUFFER_FLAG_CODEC_CONFIG. Only adding that flag for the first input buffer fixed the issue.
Related
How can I create a data frame from multiple dictionaries?
Suppose the following:
import numpy as np
import pandas as pd
Open = {'Open': np.array([86.34, 84.04, 79.06, 78.46, 75.85, 80.78, 79.66, 80.67, 82.32,80.1 , 77.63, 77. , 79.15, 76.32, 77. , 77.11, 77.04, 79.74,79.92, 79.09])}
High = {'High': np.array([86.45, 84.24, 80.29, 79.11, 79.98, 80.98, 80.57, 82.18, 83.25,81.25, 78.28, 79.2 , 79.19, 77.55, 79. , 77.5 , 81.93, 81.04,82.48, 86.74])}
Low = {'Low': np.array([83.15, 79.07, 75.59, 76.99, 74.78, 77.45, 78.48, 80.11, 80.35, 77. , 71.96, 76.15, 76.73, 75.83, 76.11, 73.46, 76.55, 78.7 ,77.65, 78.47])}
Close = {'Close': np.array([84.02, 79.17, 77.28, 77.56, 79.24, 79.86, 79.91, 82.03, 81.83,77.63, 76.19, 79.13, 76.85, 76.98, 78.31, 77.49, 81.65, 80.57,77.92, 85.51])}
index = pd.date_range('2021-1-1',periods=20)
I'm able to create a dataframe from one dictionary as evidenced by the below:
df = pd.DataFrame(Open, index = index)
However, I'm unable to extend this syntax using a list of dictionaries and get:
df = pd.DataFrame([Open, High, Low, Close] index = index)
ValueError: Shape of passed values is (4,4) indices imply (20,4)
How can I construct a dataframe from multiple dictionaries where each column is a dictionary?
You merge multiple dict into one
df = pd.DataFrame(dict(Open, **High, **Low, **Close), index = index)
df
Open High Low Close
2021-01-01 86.34 86.45 83.15 84.02
2021-01-02 84.04 84.24 79.07 79.17
2021-01-03 79.06 80.29 75.59 77.28
2021-01-04 78.46 79.11 76.99 77.56
2021-01-05 75.85 79.98 74.78 79.24
2021-01-06 80.78 80.98 77.45 79.86
2021-01-07 79.66 80.57 78.48 79.91
2021-01-08 80.67 82.18 80.11 82.03
2021-01-09 82.32 83.25 80.35 81.83
2021-01-10 80.10 81.25 77.00 77.63
2021-01-11 77.63 78.28 71.96 76.19
2021-01-12 77.00 79.20 76.15 79.13
2021-01-13 79.15 79.19 76.73 76.85
2021-01-14 76.32 77.55 75.83 76.98
2021-01-15 77.00 79.00 76.11 78.31
2021-01-16 77.11 77.50 73.46 77.49
2021-01-17 77.04 81.93 76.55 81.65
2021-01-18 79.74 81.04 78.70 80.57
2021-01-19 79.92 82.48 77.65 77.92
2021-01-20 79.09 86.74 78.47 85.51
The below code in python 3 returns as
fro val in A:
print (val)
Returns:
[(b'3 (RFC822 {821}', b'MIME-Version: 1.0\r\nDate: Sun, 2 Feb 2020
22:12:19 +0530\r\nMessage-ID:
\r\nSubject:
code\r\nFrom: abc >\r\nTo: abc \r\nContent-Type:
multipart/alternative;
boundary="43434343"\r\n\r\n--0000000000008ecb2e059d9a7dfe\r\nContent-Type:
text/plain; charset="UTF-8"\r\n\r\n1. 4549 3867 6. 1755 6816\r\n2.
3068 0287 7. 8557 7000\r\n3. 3827 1727 8. 4177 1609\r\n4. 5093 4909 9.
9799 3366\r\n5. 1069 7992 10. 5141
2029\r\n\r\n--0000000000008ecb2e059d9a7dfe\r\nContent-Type: text/html;
charset="UTF-8"\r\n\r\ntest
code\r\n\r\n--0000000000008ecb2e059d9a7dfe--'), b')']
whereas for
for val in A:
for v in val:
print (v)
returns:
b'3 (RFC822 {821}' b'MIME-Version: 1.0\r\nDate: Sun, 2 Feb 2020
22:12:19 +0530\r\nMessage-ID:
\r\nSubject:
code\r\nFrom: >\r\nTo: \r\nContent-Type: multipart/alternative;
boundary="0000000000008ecb2e059d9a7dfe"\r\n\r\n--0000000000008ecb2e059d9a7dfe\r\nContent-Type:
text/plain; charset="UTF-8"\r\n\r\n1. 4549 3867 6. 1755 6816\r\n2.
3068 0287 7. 8557 7000\r\n3. 3827 1727 8. 4177 1609\r\n4. 5093 4909 9.
9799 3366\r\n5. 1069 7992 10. 5141
2029\r\n\r\n--0000000000008ecb2e059d9a7dfe\r\nContent-Type: text/html;
charset="UTF-8"\r\n\r\n\r\n\r\n--0000000000008ecb2e059d9a7dfe--' 41
I dont understand why i am getting ASCII Value of ')' i.e 41 and how can i just read it as ')'
It would appear that when you iterate of a bytes object, it yields a decimal value for each byte in it:
eg:
>>> phrase = b'Hello, world!'
>>> chars = list(phrase)
>>> chars
[72, 101, 108, 108, 111, 44, 32, 119, 111, 114, 108, 100, 33]
With your example, you only have one character, so it's only one number that's printed. The reason none of the other bytes strings are printed as integers is because they're incased in an extra tuple, so instead it's the tuple that's iterated over.
Fixing your code:
A = [
(
b'3 (RFC822 {821}',
b'MIME-Version: 1.0\r\nDate: Sun, 2 Feb 2020 22:12:19 +0530\r\nMessage-ID: \r\nSubject: code\r\nFrom: abc >\r\nTo: abc \r\nContentType: multipart/alternative; boundary="43434343"\r\n\r\n--0000000000008ecb2e059d9a7dfe\r\nContent-Type: text/plain; charset="UTF-8"\r\n\r\n1. 4549 3867 6. 1755 6816\r\n2. 3068 0287 7. 8557 7000\r\n3. 3827 1727 8. 4177 1609\r\n4. 5093 4909 9. 9799 3366\r\n5. 1069 7992 10. 5141 2029\r\n\r\n--0000000000008ecb2e059d9a7dfe\r\nContent-Type: text/html; charset="UTF-8"\r\n\r\ntest code\r\n\r\n--0000000000008ecb2e059d9a7dfe--'
),
b')'
]
for val in A:
if isinstance(val, bytes):
print(val)
else:
for v in val:
print (v)
Output:
b'3 (RFC822 {821}'
b'MIME-Version: 1.0\r\nDate: Sun, 2 Feb 2020 22:12:19 +0530\r\nMessage-ID: \r\nSubject: code\r\nFrom: abc >\r\nTo: abc \r\nContentType: multipart/alternative; boundary="43434343"\r\n\r\n--0000000000008ecb2e059d9a7dfe\r\nContent-Type: text/plain; charset="UTF-8"\r\n\r\n1. 4549 3867 6. 1755 6816\r\n2. 3068 0287 7. 8557 7000\r\n3. 3827 1727 8. 4177 1609\r\n4. 5093 4909 9. 9799 3366\r\n5. 1069 7992 10. 5141 2029\r\n\r\n--0000000000008ecb2e059d9a7dfe\r\nContent-Type: text/html; charset="UTF-8"\r\n\r\ntest code\r\n\r\n--0000000000008ecb2e059d9a7dfe--'
b')'
Basically, if the object that we've encountered in A is a bytes-object, we print it as it is. Otherwise (eg if it's a tuple), we iterate over it, and print its items.
Here is a sample of the code:
data.timestamp = pd.to_datetime(data.timestamp, infer_datetime_format = True, utc = True)
data.timestamp.dtype
CategoricalDtype(categories=['2016-01-10 06:00:00+00:00', '2016-01-10 07:00:00+00:00',
'2016-01-10 08:00:00+00:00', '2016-01-10 09:00:00+00:00',
'2016-01-10 10:00:00+00:00', '2016-01-10 11:00:00+00:00',
'2016-01-10 12:00:00+00:00', '2016-01-10 13:00:00+00:00',
'2016-01-10 14:00:00+00:00', '2016-01-10 15:00:00+00:00',
...
'2016-12-31 13:00:00+00:00', '2016-12-31 14:00:00+00:00',
'2016-12-31 15:00:00+00:00', '2016-12-31 16:00:00+00:00',
'2016-12-31 17:00:00+00:00', '2016-12-31 18:00:00+00:00',
'2016-12-31 19:00:00+00:00', '2016-12-31 20:00:00+00:00',
'2016-12-31 21:00:00+00:00', '2016-12-31 23:00:00+00:00'],
ordered=False)
How can I solve this issue?
data.timestamp = pd.to_datetime(data.timestamp, infer_datetime_format = True, utc = True).astype('datetime64[ns]')
This worked.
I have an object, it seems to be a window object, EWM [com=9.5,min_periods=0,adjust=True,ignore_na=False,axis=0], it was created from a dataframe predictions_df_list["prices"] to be a one with dates as index and exponential weighted average of prices as values. I wanted to add it to a dataframe: predictions_df_list['ewma']. Yet it raised a NotImplementedError in inferring:
---------------------------------------------------------------------------
NotImplementedError Traceback (most recent call last)
<ipython-input-21-b1286fe39d1c> in <module>
---> 59 predictions_df_list['ewma'] = pd.DataFrame.ewm(predictions_df_list["prices"], span=20) #pd.DataFrame.ewma
60 predictions_df_list['actual_value'] = test['prices']
61 predictions_df_list['actual_value_ewma'] = pd.DataFrame.ewm(predictions_df_list["actual_value"], span=20)
C:\ProgramData\Anaconda3\lib\site-packages\pandas\core\frame.py in __setitem__(self, key, value)
3117 else:
3118 # set column
-> 3119 self._set_item(key, value)
3120
3121 def _setitem_slice(self, key, value):
C:\ProgramData\Anaconda3\lib\site-packages\pandas\core\frame.py in _set_item(self, key, value)
3192
3193 self._ensure_valid_index(value)
-> 3194 value = self._sanitize_column(key, value)
3195 NDFrame._set_item(self, key, value)
3196
C:\ProgramData\Anaconda3\lib\site-packages\pandas\core\frame.py in _sanitize_column(self, key, value, broadcast)
3385 value = _sanitize_index(value, self.index, copy=False)
3386
-> 3387 elif isinstance(value, Index) or is_sequence(value):
3388 from pandas.core.series import _sanitize_index
3389
C:\ProgramData\Anaconda3\lib\site-packages\pandas\core\dtypes\inference.py in is_sequence(obj)
470
471 try:
--> 472 iter(obj) # Can iterate over it.
473 len(obj) # Has a length associated with it.
474 return not isinstance(obj, string_and_binary_types)
C:\ProgramData\Anaconda3\lib\site-packages\pandas\core\window.py in __iter__(self)
184 def __iter__(self):
185 url = 'https://github.com/pandas-dev/pandas/issues/11704'
--> 186 raise NotImplementedError('See issue #11704 {url}'.format(url=url))
187
188 def _get_index(self, index=None):
NotImplementedError: See issue #11704 https://github.com/pandas-dev/pandas/issues/11704
When looking for documentation on window objects which seems that window objects are Python2 objects. Anyway here is predictions_df_list["prices"] which I am working with for a reproducing the error :
2007-11-01 14021.1
2007-11-02 13825.1
2007-11-03 13533.1
2007-11-04 14021.1
2007-11-05 13345.1
2007-11-06 12578.1
2007-11-07 14021.1
2007-11-08 13533.1
2007-11-09 12678.1
2007-11-10 12578.1
2007-11-11 14021.1
2007-11-12 13825.1
2007-11-13 13533.1
2007-11-14 12661.1
2007-11-15 13320.1
2007-11-16 12678.1
2007-11-17 12775.1
2007-11-18 13533.1
2007-11-19 13868.1
2007-11-20 12581.1
2007-11-21 13345.1
2007-11-22 13533.1
2007-11-23 12678.1
2007-11-24 13533.1
2007-11-25 12684.1
2007-11-26 13825.1
2007-11-27 14021.1
2007-11-28 14021.1
2007-11-29 12678.1
2007-11-30 12578.1
...
2007-12-02 13320.1
2007-12-03 12661.1
2007-12-04 13533.1
2007-12-05 12578.1
2007-12-06 13533.1
2007-12-07 13533.1
2007-12-08 14021.1
2007-12-09 12639.1
2007-12-10 12661.1
2007-12-11 13345.1
2007-12-12 12578.1
2007-12-13 14021.1
2007-12-14 13345.1
2007-12-15 13533.1
2007-12-16 12895.1
2007-12-17 13686.1
2007-12-18 14052.1
2007-12-19 14021.1
2007-12-20 13686.1
2007-12-21 12730.1
2007-12-22 13686.1
2007-12-23 12586.1
2007-12-24 12741.1
2007-12-25 12678.1
2007-12-26 13533.1
2007-12-27 12775.1
2007-12-28 12578.1
2007-12-29 12661.1
2007-12-30 12895.1
2007-12-31 12639.1
Freq: D, Name: prices, Length: 61, dtype: float64
Your ewma values can be found by using the EMA object you have and calling .mean() on it.
df['ewm'] = df['values'].ewm(alpha=0.001).mean()
I have the following data frame my_df:
name date A_score B_score
------------------------------------------
John 2017-01-01 5 6
John 2017-01-10 10 8
John 2017-02-04 3 5
Andy 2017-01-25 8 9
Andy 2017 02-05 7 1
Andy 2017-02-12 9 9
For each name, we want to find the absolute delta of A_score and B_score. The absolute delta is defined as the absolute value difference between the earliest date and the second earliest date.
The resulting data frame should be like:
name A_score_result B_score_result
----------------------------------------------
John 5 2
Andy 1 8
To achieve this, I tried:
new_df = my_df.groupby('name').apply(lambda x:myFun(x))
and
new_df = my_df.groupby('name').agg(['myFun'])
where myFun is:
def myFun(x):
y = x[2]-x[1]
return y
However, both approaches have errors like below:
/usr/local/lib/python3.4/dist-packages/pandas/core/frame.py in __getitem__(self, key)
2057 return self._getitem_multilevel(key)
2058 else:
-> 2059 return self._getitem_column(key)
2060
2061 def _getitem_column(self, key):
/usr/local/lib/python3.4/dist-packages/pandas/core/frame.py in _getitem_column(self, key)
2064 # get column
2065 if self.columns.is_unique:
-> 2066 return self._get_item_cache(key)
2067
2068 # duplicate columns & possible reduce dimensionality
/usr/local/lib/python3.4/dist-packages/pandas/core/generic.py in _get_item_cache(self, item)
1384 res = cache.get(item)
1385 if res is None:
-> 1386 values = self._data.get(item)
1387 res = self._box_item_values(item, values)
1388 cache[item] = res
/usr/local/lib/python3.4/dist-packages/pandas/core/internals.py in get(self, item, fastpath)
3541
3542 if not isnull(item):
-> 3543 loc = self.items.get_loc(item)
3544 else:
3545 indexer = np.arange(len(self.items))[isnull(self.items)]
/usr/local/lib/python3.4/dist-packages/pandas/indexes/base.py in get_loc(self, key, method, tolerance)
2134 return self._engine.get_loc(key)
2135 except KeyError:
-> 2136 return self._engine.get_loc(self._maybe_cast_indexer(key))
2137
2138 indexer = self.get_indexer([key], method=method, tolerance=tolerance)
pandas/index.pyx in pandas.index.IndexEngine.get_loc (pandas/index.c:4145)()
pandas/index.pyx in pandas.index.IndexEngine.get_loc (pandas/index.c:4009)()
pandas/src/hashtable_class_helper.pxi in pandas.hashtable.PyObjectHashTable.get_item (pandas/hashtable.c:13166)()
pandas/src/hashtable_class_helper.pxi in pandas.hashtable.PyObjectHashTable.get_item (pandas/hashtable.c:13120)()
KeyError: 0
Any suggestion on how to fix this problem? Thanks a lot!
Try this:
In [358]: df.drop('date',1).groupby('name').agg(lambda x: abs(x.iloc[1] - x.iloc[0]))
Out[358]:
A_score B_score
name
Andy 1 8
John 5 2