I don't know why na_values is not changing the values with "$-" to NaN. I have manually entered the $- in the file and there are no spaces.
import pandas as pd
df=pd.read_csv('discounted_products.csv',na_values = ['$-'])
df.head()
enter image description here
Please help here.
It could be because pandas works with regex by default on string methods (albeit it is not mentioned in the specific read_csv documentation). Try
na_values = [r'$-']
Update
It worked fine for me
from io import StringIO
df = pd.read_csv(StringIO(
'''a,b
test,$-'''), na_values=[r'$-'])
print(df)
a b
0 test NaN
Related
I am trying to change all negative values to 0 in excel files.
However, it seems like the pandas skips the first row.
Please help me with preventing skip issue! Thank you.
Here is the code:
# importing pandas module
import pandas as pd
import numpy as np
import csv
from pandas import DataFrame
df = pd.read_csv("FAPI-N2-rere_2D_modified.csv")
df[df < 0 ] = 0
df.to_csv('FAPI-N2-rere_2D_modified2.csv')
==========================================================
I have tried to add some codes into the above,
# importing pandas module
import pandas as pd
import numpy as np
import csv
from pandas import DataFrame
df = pd.read_csv("FAPI-N2-rere_2D_modified.csv", **header = None**)
df[df < 0 ] = 0
df.to_csv('FAPI-N2-rere_2D_modified2.csv')
However, i keep getting the typeerror:
TypeError: '<' not supported between instances of 'str' and 'int'
I would be so much appreciated if anyone could please help me.
Thank you so much!
Some columns of your dataframe contains strings.
If their content is numeric you can try to convert them into numeric datatype, here it is explained how to do that:
Change column type in pandas
If you have columns containing strings, you need to replace your negative values only on numeric columns.
I am trying to run the below script to add to columns to the left of a file; however it keeps giving me
valueError: header must be integer or list of integers
Below is my code:
import pandas as pd
import numpy as np
read_file = pd.read_csv("/home/ex.csv",header='true')
df=pd.DataFrame(read_file)
def add_col(x):
df.insert(loc=0, column='Creation_DT', value=pd.to_datetime('today'))
df.insert(loc=1, column='Creation_By', value="Sean")
df.to_parquet("/home/sample.parquet")
add_col(df)
Any ways to make the creation_dt column a string?
According to pandas docs header is row number(s) to use as the column names, and the start of the data and must be int or list of int. So you have to pass header=0 to read_csv method.
https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.read_csv.html
Also, pandas automatically creates dataframe from read file, you don't need to do it additionally. Use just
df = pd.read_csv("/home/ex.csv", header=0)
You can try:
import pandas as pd
import numpy as np
read_file = pd.read_csv("/home/ex.csv")
df=pd.DataFrame(read_file)
def add_col(x):
df.insert(loc=0, column='Creation_DT', value=str(pd.to_datetime('today')))
df.insert(loc=1, column='Creation_By', value="Sean")
df.to_parquet("/home/sample.parquet")
add_col(df)
The script:
import csv
import pandas as pd
df2 = pd.read_csv("aapl.csv", header= 0 , encoding = 'latin-1')
print(df2)
result:
enter image description here
As you can see when printing where there is CSV formula I get the formula instead of the value...
any solutions?
you can specify the column type
import openpyxl, csv
import pandas as pd
df2 = pd.read_csv("aapl.csv", header= 0 , encoding = 'latin-1', dtype={"upper": int,"lower": int})
print(df2)
hope this will help you
I imported a file from spss, (sav file), however, the titles of my columns
appear as integers instead of strings. Is there a way to fix it? Below is the code I used....I would apreciate any help!
import fnmatch
import sys # import sys
import os
import pandas as pd #pandas importer
import savReaderWriter as spss # to import file from SPSS
import io #importing io
import codecs #to resolve the UTF-8 unicode
with spss.SavReader('file_name.sav') as reader: #Should I add "Np"
records = reader.all()
with codecs.open('file_name.sav', "r",encoding='utf-8', errors='strict')
as fdata: # Not sure if the problems resides on this line
df = pd.DataFrame(records)
df.head()
Wondering whether there is a way to actually convert the titles from numbers to strings. It has happened as if it were excel, but excel has an easy fix for that.
Thanks in advance!
After you have created the DataFrame, you can use df.columns = df.columns.map(str) to change the column headers to strings.
New coder here, trying to run some t-tests in Python 3.6. Right now, to run my t-tests between my 2 data sets, I have been doing the following:
import plotly.plotly as py
import plotly.graph_objs as go
from plotly.tools import FigureFactory as FF
import numpy as np
import pandas as pd
import scipy
from scipy import stats
long_term_survivor_GENE1 = [-0.38,-0.99,-1.04,0.1, etc..]
short_term_survivor_GENE1 = [0.32, 0.33,0.96, etc...]
stats.ttest_ind(long_term_survivor_GENE1,short_term_survivor_GENE1)
Which requires me to manually enter the values for each column of both data sets for each specific gene (GENE1 in this case). Is there any way to be able to call for the values from the data set so that Python can just read the values without me typing them out myself? For example, some way that I can just say:
long_term_survivor_GENE1 = ##call values from GENE1 column from dataset 1##
short_term_survivor_GENE1 = ## call values from GENE1 column from dataset 2##
Thanks for any help, and sorry that I'm not very well-versed in this stuff. Appreciate any feedback/tips. If you have any other questions, please let me know!
If you've shoved your data into the columns of a pandas dataframe then it might be as easy as this.
>>> import pandas as pd
>>> long_term_survivor_GENE1 = [-0.38,-0.99,-1.04,0.1]
>>> short_term_survivor_GENE1 = [0.32, 0.33,0.96, 0.56]
>>> df = pd.DataFrame({'long_term_survivor_GENE1': long_term_survivor_GENE1, 'short_term_survivor_GENE1': short_term_survivor_GENE1})
>>> from scipy import stats
>>> stats.ttest_ind(df['long_term_survivor_GENE1'], df['short_term_survivor_GENE1'])
Ttest_indResult(statistic=-3.615804684179662, pvalue=0.011153077626049458)
It might be a good idea to review the statistics behind this though. If you haven't already got them in a dataframe then have a look for some of the many answers here on SO about using read_csv for assistance.