Cannot use pip install pytorch-pretrained-bert - pytorch
I was trying to use "pip install pytorch-pretrained-bert", but I met a unseen problem. It says that the version of botocore is not satisfied. They need botocore<1.18.0,>=1.17.54. However, according to the rest information, I found that the latest version of botocore is 1.17.53, it never actually reached 1.17.54. So I'm wondering how to solve this problem. Thanks for your help!
The error message is as follows,
ERROR: Could not find a version that satisfies the requirement botocore<1.18.0,>=1.17.54 (from boto3->pytorch-pretrained-bert) (from versions: 0.4.1, 0.4.2, 0.5.0, 0.5.1, 0.5.2, 0.5.3, 0.5.4, 0.6.0, 0.7.0, 0.8.0, 0.8.1, 0.8.2, 0.8.3, 0.9.0, 0.9.1, 0.9.2, 0.10.0, 0.11.0, 0.12.0, 0.13.0, 0.13.1, 0.14.0, 0.15.0, 0.15.1, 0.16.0, 0.17.0, 0.18.0, 0.19.0, 0.20.0, 0.21.0, 0.22.0, 0.23.0, 0.24.0, 0.25.0, 0.26.0, 0.27.0, 0.28.0, 0.29.0, 0.30.0, 0.31.0, 0.32.0, 0.33.0, 0.34.0, 0.35.0, 0.36.0, 0.37.0, 0.38.0, 0.39.0, 0.40.0, 0.41.0, 0.42.0, 0.43.0, 0.44.0, 0.45.0, 0.46.0, 0.47.0, 0.48.0, 0.49.0, 0.50.0, 0.51.0, 0.52.0, 0.53.0, 0.54.0, 0.55.0, 0.56.0, 0.57.0, 0.58.0, 0.59.0, 0.60.0, 0.61.0, 0.62.0, 0.63.0, 0.64.0, 0.65.0, 0.66.0, 0.67.0, 0.68.0, 0.69.0, 0.70.0, 0.71.0, 0.72.0, 0.73.0, 0.74.0, 0.75.0, 0.76.0, 0.77.0, 0.78.0, 0.79.0, 0.80.0, 0.81.0, 0.82.0, 0.83.0, 0.84.0, 0.85.0, 0.86.0, 0.87.0, 0.88.0, 0.89.0, 0.90.0, 0.91.0, 0.92.0, 0.93.0, 0.94.0, 0.95.0, 0.96.0, 0.97.0, 0.98.0, 0.99.0, 0.100.0, 0.101.0, 0.102.0, 0.103.0, 0.104.0, 0.105.0, 0.106.0, 0.107.0, 0.108.0, 0.109.0, 1.0.0a1, 1.0.0a2, 1.0.0a3, 1.0.0b1, 1.0.0b2, 1.0.0b3, 1.0.0rc1, 1.0.0, 1.0.1, 1.1.0, 1.1.1, 1.1.2, 1.1.3, 1.1.4, 1.1.5, 1.1.6, 1.1.7, 1.1.8, 1.1.9, 1.1.10, 1.1.11, 1.1.12, 1.2.0, 1.2.1, 1.2.2, 1.2.3, 1.2.4, 1.2.5, 1.2.6, 1.2.7, 1.2.8, 1.2.9, 1.2.10, 1.2.11, 1.3.0, 1.3.1, 1.3.2, 1.3.3, 1.3.4, 1.3.5, 1.3.6, 1.3.7, 1.3.8, 1.3.9, 1.3.10, 1.3.11, 1.3.12, 1.3.13, 1.3.14, 1.3.15, 1.3.16, 1.3.17, 1.3.18, 1.3.19, 1.3.20, 1.3.21, 1.3.22, 1.3.23, 1.3.24, 1.3.25, 1.3.26, 1.3.27, 1.3.28, 1.3.29, 1.3.30, 1.4.0, 1.4.1, 1.4.2, 1.4.3, 1.4.4, 1.4.5, 1.4.6, 1.4.7, 1.4.8, 1.4.9, 1.4.10, 1.4.11, 1.4.12, 1.4.13, 1.4.14, 1.4.15, 1.4.16, 1.4.17, 1.4.18, 1.4.19, 1.4.20, 1.4.21, 1.4.22, 1.4.23, 1.4.24, 1.4.25, 1.4.26, 1.4.27, 1.4.28, 1.4.29, 1.4.30, 1.4.31, 1.4.32, 1.4.33, 1.4.34, 1.4.35, 1.4.36, 1.4.37, 1.4.38, 1.4.39, 1.4.40, 1.4.41, 1.4.42, 1.4.43, 1.4.44, 1.4.46, 1.4.47, 1.4.48, 1.4.49, 1.4.50, 1.4.51, 1.4.52, 1.4.53, 1.4.54, 1.4.55, 1.4.56, 1.4.57, 1.4.58, 1.4.59, 1.4.60, 1.4.61, 1.4.62, 1.4.63, 1.4.64, 1.4.65, 1.4.66, 1.4.67, 1.4.68, 1.4.69, 1.4.70, 1.4.71, 1.4.72, 1.4.73, 1.4.74, 1.4.75, 1.4.76, 1.4.77, 1.4.78, 1.4.79, 1.4.80, 1.4.81, 1.4.82, 1.4.83, 1.4.84, 1.4.85, 1.4.86, 1.4.87, 1.4.88, 1.4.89, 1.4.90, 1.4.91, 1.4.92, 1.4.93, 1.5.0, 1.5.1, 1.5.2, 1.5.3, 1.5.4, 1.5.5, 1.5.6, 1.5.7, 1.5.8, 1.5.9, 1.5.10, 1.5.11, 1.5.12, 1.5.13, 1.5.14, 1.5.15, 1.5.16, 1.5.17, 1.5.18, 1.5.19, 1.5.20, 1.5.21, 1.5.22, 1.5.23, 1.5.24, 1.5.25, 1.5.26, 1.5.27, 1.5.28, 1.5.29, 1.5.30, 1.5.31, 1.5.32, 1.5.33, 1.5.34, 1.5.35, 1.5.36, 1.5.37, 1.5.38, 1.5.39, 1.5.40, 1.5.41, 1.5.42, 1.5.43, 1.5.44, 1.5.45, 1.5.46, 1.5.47, 1.5.48, 1.5.49, 1.5.50, 1.5.51, 1.5.52, 1.5.53, 1.5.54, 1.5.55, 1.5.56, 1.5.57, 1.5.58, 1.5.59, 1.5.60, 1.5.61, 1.5.62, 1.5.63, 1.5.64, 1.5.65, 1.5.66, 1.5.67, 1.5.68, 1.5.69, 1.5.70, 1.5.71, 1.5.72, 1.5.73, 1.5.74, 1.5.75, 1.5.76, 1.5.77, 1.5.78, 1.5.79, 1.5.80, 1.5.81, 1.5.82, 1.5.83, 1.5.84, 1.5.85, 1.5.86, 1.5.87, 1.5.88, 1.5.89, 1.5.90, 1.5.91, 1.5.92, 1.5.93, 1.5.94, 1.5.95, 1.6.0, 1.6.1, 1.6.2, 1.6.3, 1.6.4, 1.6.5, 1.6.6, 1.6.7, 1.6.8, 1.7.0, 1.7.1, 1.7.2, 1.7.3, 1.7.4, 1.7.5, 1.7.6, 1.7.7, 1.7.8, 1.7.9, 1.7.10, 1.7.11, 1.7.12, 1.7.13, 1.7.14, 1.7.15, 1.7.16, 1.7.17, 1.7.18, 1.7.19, 1.7.20, 1.7.21, 1.7.22, 1.7.23, 1.7.24, 1.7.25, 1.7.26, 1.7.27, 1.7.28, 1.7.29, 1.7.30, 1.7.31, 1.7.32, 1.7.33, 1.7.34, 1.7.35, 1.7.36, 1.7.37, 1.7.38, 1.7.39, 1.7.40, 1.7.41, 1.7.42, 1.7.43, 1.7.44, 1.7.45, 1.7.46, 1.7.47, 1.7.48, 1.8.0, 1.8.1, 1.8.2, 1.8.3, 1.8.4, 1.8.5, 1.8.6, 1.8.7, 1.8.8, 1.8.9, 1.8.10, 1.8.11, 1.8.12, 1.8.13, 1.8.14, 1.8.15, 1.8.16, 1.8.17, 1.8.18, 1.8.19, 1.8.20, 1.8.21, 1.8.22, 1.8.23, 1.8.24, 1.8.25, 1.8.26, 1.8.27, 1.8.28, 1.8.29, 1.8.30, 1.8.31, 1.8.32, 1.8.33, 1.8.34, 1.8.35, 1.8.36, 1.8.37, 1.8.38, 1.8.39, 1.8.40, 1.8.41, 1.8.42, 1.8.43, 1.8.44, 1.8.45, 1.8.46, 1.8.47, 1.8.48, 1.8.49, 1.8.50, 1.9.0, 1.9.1, 1.9.2, 1.9.3, 1.9.4, 1.9.5, 1.9.6, 1.9.7, 1.9.8, 1.9.9, 1.9.10, 1.9.11, 1.9.12, 1.9.13, 1.9.14, 1.9.15, 1.9.16, 1.9.17, 1.9.18, 1.9.19, 1.9.20, 1.9.21, 1.9.22, 1.9.23, 1.10.0, 1.10.1, 1.10.2, 1.10.3, 1.10.4, 1.10.5, 1.10.6, 1.10.7, 1.10.8, 1.10.9, 1.10.10, 1.10.11, 1.10.12, 1.10.13, 1.10.14, 1.10.15, 1.10.16, 1.10.17, 1.10.18, 1.10.19, 1.10.20, 1.10.21, 1.10.22, 1.10.23, 1.10.24, 1.10.25, 1.10.26, 1.10.27, 1.10.28, 1.10.29, 1.10.30, 1.10.31, 1.10.32, 1.10.33, 1.10.34, 1.10.35, 1.10.36, 1.10.37, 1.10.38, 1.10.39, 1.10.40, 1.10.41, 1.10.42, 1.10.43, 1.10.44, 1.10.45, 1.10.46, 1.10.47, 1.10.48, 1.10.49, 1.10.50, 1.10.51, 1.10.52, 1.10.53, 1.10.54, 1.10.55, 1.10.56, 1.10.57, 1.10.58, 1.10.59, 1.10.60, 1.10.61, 1.10.62, 1.10.63, 1.10.64, 1.10.65, 1.10.66, 1.10.67, 1.10.68, 1.10.69, 1.10.70, 1.10.71, 1.10.72, 1.10.73, 1.10.74, 1.10.75, 1.10.76, 1.10.77, 1.10.78, 1.10.79, 1.10.80, 1.10.81, 1.10.82, 1.10.83, 1.10.84, 1.11.0, 1.11.1, 1.11.2, 1.11.3, 1.11.4, 1.11.5, 1.11.6, 1.11.7, 1.11.8, 1.11.9, 1.12.0, 1.12.1, 1.12.2, 1.12.3, 1.12.4, 1.12.5, 1.12.6, 1.12.7, 1.12.8, 1.12.9, 1.12.10, 1.12.11, 1.12.12, 1.12.13, 1.12.14, 1.12.15, 1.12.16, 1.12.17, 1.12.18, 1.12.19, 1.12.20, 1.12.21, 1.12.22, 1.12.23, 1.12.24, 1.12.25, 1.12.26, 1.12.27, 1.12.28, 1.12.29, 1.12.30, 1.12.31, 1.12.32, 1.12.33, 1.12.34, 1.12.35, 1.12.36, 1.12.37, 1.12.38, 1.12.39, 1.12.40, 1.12.41, 1.12.42, 1.12.43, 1.12.44, 1.12.45, 1.12.46, 1.12.47, 1.12.48, 1.12.49, 1.12.50, 1.12.51, 1.12.52, 1.12.53, 1.12.54, 1.12.55, 1.12.56, 1.12.57, 1.12.58, 1.12.59, 1.12.60, 1.12.61, 1.12.62, 1.12.63, 1.12.64, 1.12.65, 1.12.66, 1.12.67, 1.12.68, 1.12.69, 1.12.70, 1.12.71, 1.12.72, 1.12.73, 1.12.74, 1.12.75, 1.12.76, 1.12.77, 1.12.78, 1.12.79, 1.12.80, 1.12.81, 1.12.82, 1.12.83, 1.12.84, 1.12.85, 1.12.86, 1.12.87, 1.12.88, 1.12.89, 1.12.90, 1.12.91, 1.12.92, 1.12.93, 1.12.94, 1.12.95, 1.12.96, 1.12.97, 1.12.98, 1.12.99, 1.12.100, 1.12.101, 1.12.102, 1.12.103, 1.12.104, 1.12.105, 1.12.106, 1.12.107, 1.12.108, 1.12.109, 1.12.110, 1.12.111, 1.12.112, 1.12.113, 1.12.114, 1.12.115, 1.12.116, 1.12.117, 1.12.118, 1.12.119, 1.12.120, 1.12.121, 1.12.122, 1.12.123, 1.12.124, 1.12.125, 1.12.126, 1.12.127, 1.12.128, 1.12.129, 1.12.130, 1.12.131, 1.12.132, 1.12.133, 1.12.134, 1.12.135, 1.12.136, 1.12.137, 1.12.138, 1.12.139, 1.12.140, 1.12.141, 1.12.142, 1.12.143, 1.12.144, 1.12.145, 1.12.146, 1.12.147, 1.12.148, 1.12.149, 1.12.150, 1.12.151, 1.12.152, 1.12.153, 1.12.154, 1.12.155, 1.12.156, 1.12.157, 1.12.158, 1.12.159, 1.12.160, 1.12.161, 1.12.162, 1.12.163, 1.12.164, 1.12.165, 1.12.166, 1.12.167, 1.12.168, 1.12.169, 1.12.170, 1.12.171, 1.12.172, 1.12.173, 1.12.174, 1.12.175, 1.12.176, 1.12.177, 1.12.178, 1.12.179, 1.12.180, 1.12.181, 1.12.182, 1.12.183, 1.12.184, 1.12.185, 1.12.186, 1.12.187, 1.12.188, 1.12.189, 1.12.190, 1.12.191, 1.12.192, 1.12.193, 1.12.194, 1.12.195, 1.12.196, 1.12.197, 1.12.198, 1.12.199, 1.12.200, 1.12.201, 1.12.202, 1.12.203, 1.12.204, 1.12.205, 1.12.206, 1.12.207, 1.12.208, 1.12.209, 1.12.210, 1.12.211, 1.12.212, 1.12.213, 1.12.214, 1.12.215, 1.12.216, 1.12.217, 1.12.218, 1.12.219, 1.12.220, 1.12.221, 1.12.222, 1.12.223, 1.12.224, 1.12.225, 1.12.226, 1.12.227, 1.12.228, 1.12.229, 1.12.230, 1.12.231, 1.12.232, 1.12.233, 1.12.234, 1.12.235, 1.12.236, 1.12.237, 1.12.238, 1.12.239, 1.12.240, 1.12.241, 1.12.242, 1.12.243, 1.12.244, 1.12.245, 1.12.246, 1.12.247, 1.12.248, 1.12.249, 1.12.250, 1.12.251, 1.12.252, 1.12.253, 1.13.0, 1.13.1, 1.13.2, 1.13.3, 1.13.4, 1.13.5, 1.13.6, 1.13.7, 1.13.8, 1.13.9, 1.13.10, 1.13.11, 1.13.12, 1.13.13, 1.13.14, 1.13.15, 1.13.16, 1.13.17, 1.13.18, 1.13.19, 1.13.20, 1.13.21, 1.13.22, 1.13.23, 1.13.24, 1.13.25, 1.13.26, 1.13.27, 1.13.28, 1.13.29, 1.13.30, 1.13.31, 1.13.32, 1.13.33, 1.13.34, 1.13.35, 1.13.36, 1.13.37, 1.13.38, 1.13.39, 1.13.40, 1.13.41, 1.13.42, 1.13.43, 1.13.44, 1.13.45, 1.13.46, 1.13.47, 1.13.48, 1.13.49, 1.13.50, 1.14.0, 1.14.1, 1.14.2, 1.14.3, 1.14.4, 1.14.5, 1.14.6, 1.14.7, 1.14.8, 1.14.9, 1.14.10, 1.14.11, 1.14.12, 1.14.13, 1.14.14, 1.14.15, 1.14.16, 1.14.17, 1.15.0, 1.15.1, 1.15.2, 1.15.3, 1.15.4, 1.15.5, 1.15.6, 1.15.7, 1.15.8, 1.15.9, 1.15.10, 1.15.11, 1.15.12, 1.15.13, 1.15.14, 1.15.15, 1.15.16, 1.15.17, 1.15.18, 1.15.19, 1.15.20, 1.15.21, 1.15.22, 1.15.23, 1.15.24, 1.15.25, 1.15.26, 1.15.27, 1.15.28, 1.15.29, 1.15.30, 1.15.31, 1.15.32, 1.15.33, 1.15.34, 1.15.35, 1.15.36, 1.15.37, 1.15.38, 1.15.39, 1.15.40, 1.15.41, 1.15.42, 1.15.43, 1.15.44, 1.15.45, 1.15.46, 1.15.47, 1.15.48, 1.15.49, 1.16.0, 1.16.1, 1.16.2, 1.16.3, 1.16.4, 1.16.5, 1.16.6, 1.16.7, 1.16.8, 1.16.9, 1.16.10, 1.16.11, 1.16.12, 1.16.13, 1.16.14, 1.16.15, 1.16.16, 1.16.17, 1.16.18, 1.16.19, 1.16.20, 1.16.21, 1.16.22, 1.16.23, 1.16.24, 1.16.25, 1.16.26, 1.17.0, 1.17.1, 1.17.2, 1.17.3, 1.17.4, 1.17.5, 1.17.6, 1.17.7, 1.17.8, 1.17.9, 1.17.10, 1.17.11, 1.17.12, 1.17.13, 1.17.14, 1.17.15, 1.17.16, 1.17.17, 1.17.18, 1.17.19, 1.17.20, 1.17.21, 1.17.22, 1.17.23, 1.17.24, 1.17.25, 1.17.26, 1.17.27, 1.17.28, 1.17.29, 1.17.30, 1.17.31, 1.17.32, 1.17.33, 1.17.34, 1.17.35, 1.17.36, 1.17.37, 1.17.38, 1.17.39, 1.17.40, 1.17.41, 1.17.42, 1.17.43, 1.17.44, 1.17.45, 1.17.46, 1.17.47, 1.17.48, 1.17.49, 1.17.50, 1.17.51, 1.17.52, 1.17.53)
ERROR: No matching distribution found for botocore<1.18.0,>=1.17.54 (from boto3->pytorch-pretrained-bert)
I use "conda install -c conda-forge pytorch-pretrained-bert", instead of pip install. Pretty confused about the reason behind that, but at least it's solved.
Related
Error trying to use IntelliJ IDE Database Spark Connector
I try to use the IntelliJ -> Database -> New -> Other -> Spark Connector The issue is that after selecting the option, I get the exception shared at the end of the question. I can not find documentation about the right way to use it, so I suppose that I'm doing something wrong. How to fix it or where is the documentation or step-by-step tutorial or whatever that can help? IntelliJ IDEA 2022.1.1 (Ultimate Edition) Build #IU-221.5591.52, built on May 10, 2022 Licensed to XXXXXXXXXXXXXXXXXXX Subscription is active until January 3, 2023. Runtime version: 11.0.14.1+1-b2043.45 amd64 VM: OpenJDK 64-Bit Server VM by JetBrains s.r.o. Linux 5.15.0-33-generic GC: G1 Young Generation, G1 Old Generation Memory: 4096M Cores: 16 Registry: ide.balloon.shadow.size=0 scala.erase.compiler.process.jdk.once=false Non-Bundled Plugins: Key Promoter X (2022.1.2) com.4lex4.intellij.solarized (2.4.0) com.arcticicestudio.nord.jetbrains (0.13.0) com.intellij.javafx (1.0.4) com.intellij.nativeDebug (221.5591.54) com.intellij.plugins.gnomekeymap (213.4250.391) com.intellij.plugins.xwinkeymap (211.4961.30) com.markskelton.one-dark-theme (5.6.0) commit-template-idea-plugin (1.2.0) de.docs_as_co.intellij.plugin.diagramsnet (0.1.12) net.seesharpsoft.intellij.plugins.csv (2.18.2) org.antlr.intellij.plugin (1.19.2) org.intellij.plugins.hcl (221.5591.19) org.jetbrains.plugins.rest (221.5591.19) name.kropp.intellij.makefile (221.5591.19) AWSCloudFormation (221.5591.19) com.intellij.kubernetes (221.5591.58) org.jetbrains.plugins.ruby (221.5591.52) org.nik.presentation-assistant (1.0.9) org.tbm98.flutter-riverpod-snippets (2.221.0) org.rust.lang (0.4.170.4627-221) org.sonarlint.idea (6.7.0.45926) com.intellij.guice (221.5591.19) org.jetbrains.plugins.hocon (2022.1.0) org.asciidoctor.intellij.asciidoc (0.37.19) com.jetbrains.lang.ejs (221.5591.19) mdx.js (1.0.221) Dart (221.5591.58) com.localizely.flutter-intl (1.17.3-2020.3) io.flutter (67.1.4) org.intellij.scala (2022.1.15) zielu.gittoolbox (212.9.0) Pythonid (221.5591.52) com.github.shiraji.yaemoji (1.0.7) com.intellij.bigdatatools (221.4165.63) ru.adelf.idea.dotenv (2022.1) aws.toolkit (1.43-221) com.intellij.plugins.html.instantEditing (221.5591.19) Kotlin: 221-1.6.21-release-337-IJ5591.52 Current Desktop: ubuntu:GNOME This is the exception that I get just after clicking: java.lang.AbstractMethodError: Receiver class com.intellij.bigdatatools.zeppelin.ztools.database.introspector.ZeppelinSparkIntrospector$Factory does not define or inherit an implementation of the resolved method 'abstract boolean supportsMultilevelIntrospection()' of interface com.intellij.database.introspection.DBIntrospector$Factory. at com.intellij.database.introspection.DBIntrospectorFeatures.supportsMultilevelIntrospection(DBIntrospectorFeatures.kt:11) at com.intellij.database.view.ui.DatabaseDriverOptionsPanel.<init>(DatabaseDriverOptionsPanel.java:82) at com.intellij.database.view.ui.DatabaseDriverConfigurable.createComponent(DatabaseDriverConfigurable.java:78) at com.intellij.database.view.ui.SidePanelItem.createConfigurable(SidePanelItem.java:131) at com.intellij.database.view.ui.DatabaseConfigEditorImpl.createEditorIfNeeded(DatabaseConfigEditorImpl.java:582) at com.intellij.database.view.ui.DatabaseConfigEditorImpl.setSelectedEditor(DatabaseConfigEditorImpl.java:593) at com.intellij.database.view.ui.DatabaseConfigEditorImpl.navigateTo(DatabaseConfigEditorImpl.java:376) at com.intellij.database.view.ui.DatabaseConfigEditorImpl.navigateTo(DatabaseConfigEditorImpl.java:699) at com.intellij.database.view.ui.DatabaseConfigSideTab.lambda$createSidePanel$2(DatabaseConfigSideTab.java:125) at java.desktop/javax.swing.JList.fireSelectionValueChanged(JList.java:1804) at java.desktop/javax.swing.JList$ListSelectionHandler.valueChanged(JList.java:1818) at java.desktop/javax.swing.DefaultListSelectionModel.fireValueChanged(DefaultListSelectionModel.java:219) at java.desktop/javax.swing.DefaultListSelectionModel.fireValueChanged(DefaultListSelectionModel.java:186) at java.desktop/javax.swing.DefaultListSelectionModel.setValueIsAdjusting(DefaultListSelectionModel.java:723) at java.desktop/javax.swing.JList.setValueIsAdjusting(JList.java:2152) at java.desktop/javax.swing.plaf.basic.BasicListUI$Handler.mouseReleased(BasicListUI.java:2958) at java.desktop/java.awt.AWTEventMulticaster.mouseReleased(AWTEventMulticaster.java:298) at java.desktop/java.awt.AWTEventMulticaster.mouseReleased(AWTEventMulticaster.java:297) at java.desktop/java.awt.AWTEventMulticaster.mouseReleased(AWTEventMulticaster.java:297) at java.desktop/java.awt.AWTEventMulticaster.mouseReleased(AWTEventMulticaster.java:297) at java.desktop/java.awt.AWTEventMulticaster.mouseReleased(AWTEventMulticaster.java:297) at java.desktop/java.awt.Component.processMouseEvent(Component.java:6654) at java.desktop/javax.swing.JComponent.processMouseEvent(JComponent.java:3345) at java.desktop/java.awt.Component.processEvent(Component.java:6419) at java.desktop/java.awt.Container.processEvent(Container.java:2263) at java.desktop/java.awt.Component.dispatchEventImpl(Component.java:5029) at java.desktop/java.awt.Container.dispatchEventImpl(Container.java:2321) at java.desktop/java.awt.Component.dispatchEvent(Component.java:4861) at java.desktop/java.awt.LightweightDispatcher.retargetMouseEvent(Container.java:4918) at java.desktop/java.awt.LightweightDispatcher.processMouseEvent(Container.java:4547) at java.desktop/java.awt.LightweightDispatcher.dispatchEvent(Container.java:4488) at java.desktop/java.awt.Container.dispatchEventImpl(Container.java:2307) at java.desktop/java.awt.Window.dispatchEventImpl(Window.java:2793) at java.desktop/java.awt.Component.dispatchEvent(Component.java:4861) at java.desktop/java.awt.EventQueue.dispatchEventImpl(EventQueue.java:778) at java.desktop/java.awt.EventQueue$4.run(EventQueue.java:727) at java.desktop/java.awt.EventQueue$4.run(EventQueue.java:721) at java.base/java.security.AccessController.doPrivileged(Native Method) at java.base/java.security.ProtectionDomain$JavaSecurityAccessImpl.doIntersectionPrivilege(ProtectionDomain.java:85) at java.base/java.security.ProtectionDomain$JavaSecurityAccessImpl.doIntersectionPrivilege(ProtectionDomain.java:95) at java.desktop/java.awt.EventQueue$5.run(EventQueue.java:751) at java.desktop/java.awt.EventQueue$5.run(EventQueue.java:749) at java.base/java.security.AccessController.doPrivileged(Native Method) at java.base/java.security.ProtectionDomain$JavaSecurityAccessImpl.doIntersectionPrivilege(ProtectionDomain.java:85) at java.desktop/java.awt.EventQueue.dispatchEvent(EventQueue.java:748) at com.intellij.ide.IdeEventQueue.defaultDispatchEvent(IdeEventQueue.java:898) at com.intellij.ide.IdeEventQueue.dispatchMouseEvent(IdeEventQueue.java:820) at com.intellij.ide.IdeEventQueue._dispatchEvent(IdeEventQueue.java:743) at com.intellij.ide.IdeEventQueue.lambda$dispatchEvent$6(IdeEventQueue.java:439) at com.intellij.openapi.progress.impl.CoreProgressManager.computePrioritized(CoreProgressManager.java:803) at com.intellij.ide.IdeEventQueue.lambda$dispatchEvent$7(IdeEventQueue.java:438) at com.intellij.openapi.application.TransactionGuardImpl.performActivity(TransactionGuardImpl.java:106) at com.intellij.ide.IdeEventQueue.performActivity(IdeEventQueue.java:604) at com.intellij.ide.IdeEventQueue.lambda$dispatchEvent$8(IdeEventQueue.java:436) at com.intellij.openapi.application.impl.ApplicationImpl.runIntendedWriteActionOnCurrentThread(ApplicationImpl.java:873) at com.intellij.ide.IdeEventQueue.dispatchEvent(IdeEventQueue.java:484) at java.desktop/java.awt.EventDispatchThread.pumpOneEventForFilters(EventDispatchThread.java:207) at java.desktop/java.awt.EventDispatchThread.pumpEventsForFilter(EventDispatchThread.java:128) at java.desktop/java.awt.EventDispatchThread.pumpEventsForFilter(EventDispatchThread.java:121) at java.desktop/java.awt.WaitDispatchSupport$2.run(WaitDispatchSupport.java:190) at java.desktop/java.awt.WaitDispatchSupport$4.run(WaitDispatchSupport.java:235) at java.desktop/java.awt.WaitDispatchSupport$4.run(WaitDispatchSupport.java:233) at java.base/java.security.AccessController.doPrivileged(Native Method) at java.desktop/java.awt.WaitDispatchSupport.enter(WaitDispatchSupport.java:233) at java.desktop/java.awt.Dialog.show(Dialog.java:1070) at com.intellij.openapi.ui.impl.DialogWrapperPeerImpl$MyDialog.show(DialogWrapperPeerImpl.java:702) at com.intellij.openapi.ui.impl.DialogWrapperPeerImpl.show(DialogWrapperPeerImpl.java:439) at com.intellij.openapi.ui.DialogWrapper.doShow(DialogWrapper.java:1668) at com.intellij.openapi.ui.DialogWrapper.show(DialogWrapper.java:1626) at com.intellij.database.view.ui.DataSourceManagerDialog.showDialog(DataSourceManagerDialog.java:104) at com.intellij.database.view.ui.DataSourceManagerDialog.showDialog(DataSourceManagerDialog.java:79) at com.intellij.database.view.ui.DataSourceManagerDialog.showDialog(DataSourceManagerDialog.java:73) at com.intellij.database.actions.DatabaseViewActions$AddDriver.actionPerformed(DatabaseViewActions.java:274) at com.intellij.openapi.actionSystem.ex.ActionUtil.lambda$performActionDumbAwareWithCallbacks$4(ActionUtil.java:235) at com.intellij.openapi.actionSystem.ex.ActionUtil.performDumbAwareWithCallbacks(ActionUtil.java:256) at com.intellij.openapi.actionSystem.ex.ActionUtil.performActionDumbAwareWithCallbacks(ActionUtil.java:235) at com.intellij.ui.popup.ActionPopupStep.performAction(ActionPopupStep.java:240) at com.intellij.ui.popup.ActionPopupStep.performAction(ActionPopupStep.java:249) at com.intellij.ui.popup.ActionPopupStep.performAction(ActionPopupStep.java:245) at com.intellij.ui.popup.ActionPopupStep.lambda$onChosen$1(ActionPopupStep.java:220) at com.intellij.openapi.application.TransactionGuardImpl.performActivity(TransactionGuardImpl.java:106) at com.intellij.openapi.application.TransactionGuardImpl.performUserActivity(TransactionGuardImpl.java:95) at com.intellij.ui.popup.AbstractPopup.lambda$dispose$18(AbstractPopup.java:1538) at com.intellij.util.ui.EdtInvocationManager.invokeLaterIfNeeded(EdtInvocationManager.java:101) at com.intellij.ide.IdeEventQueue.ifFocusEventsInTheQueue(IdeEventQueue.java:179) at com.intellij.ide.IdeEventQueue.executeWhenAllFocusEventsLeftTheQueue(IdeEventQueue.java:132) at com.intellij.openapi.wm.impl.FocusManagerImpl.doWhenFocusSettlesDown(FocusManagerImpl.java:172) at com.intellij.ui.popup.AbstractPopup.dispose(AbstractPopup.java:1535) at com.intellij.ui.popup.WizardPopup.dispose(WizardPopup.java:162) at com.intellij.ui.popup.list.ListPopupImpl.dispose(ListPopupImpl.java:326) at com.intellij.ui.popup.PopupFactoryImpl$ActionGroupPopup.dispose(PopupFactoryImpl.java:272) at com.intellij.openapi.util.ObjectTree.runWithTrace(ObjectTree.java:139) at com.intellij.openapi.util.ObjectTree.executeAll(ObjectTree.java:169) at com.intellij.openapi.util.Disposer.dispose(Disposer.java:219) at com.intellij.openapi.util.Disposer.dispose(Disposer.java:207) at com.intellij.ui.popup.WizardPopup.disposeAllParents(WizardPopup.java:266) at com.intellij.ui.popup.list.ListPopupImpl.handleNextStep(ListPopupImpl.java:434) at com.intellij.ui.popup.list.ListPopupImpl._handleSelect(ListPopupImpl.java:406) at com.intellij.ui.popup.list.ListPopupImpl.handleSelect(ListPopupImpl.java:361) at com.intellij.ui.popup.PopupFactoryImpl$ActionGroupPopup.handleSelect(PopupFactoryImpl.java:289) at com.intellij.ui.popup.list.ListPopupImpl$MyMouseListener.mouseReleased(ListPopupImpl.java:618) at java.desktop/java.awt.AWTEventMulticaster.mouseReleased(AWTEventMulticaster.java:298) at java.desktop/java.awt.Component.processMouseEvent(Component.java:6654) at java.desktop/javax.swing.JComponent.processMouseEvent(JComponent.java:3345) at com.intellij.ui.popup.list.ListPopupImpl$MyList.processMouseEvent(ListPopupImpl.java:694) at java.desktop/java.awt.Component.processEvent(Component.java:6419) at java.desktop/java.awt.Container.processEvent(Container.java:2263) at java.desktop/java.awt.Component.dispatchEventImpl(Component.java:5029) at java.desktop/java.awt.Container.dispatchEventImpl(Container.java:2321) at java.desktop/java.awt.Component.dispatchEvent(Component.java:4861) at java.desktop/java.awt.LightweightDispatcher.retargetMouseEvent(Container.java:4918) at java.desktop/java.awt.LightweightDispatcher.processMouseEvent(Container.java:4547) at java.desktop/java.awt.LightweightDispatcher.dispatchEvent(Container.java:4488) at java.desktop/java.awt.Container.dispatchEventImpl(Container.java:2307) at java.desktop/java.awt.Window.dispatchEventImpl(Window.java:2793) at java.desktop/java.awt.Component.dispatchEvent(Component.java:4861) at java.desktop/java.awt.EventQueue.dispatchEventImpl(EventQueue.java:778) at java.desktop/java.awt.EventQueue$4.run(EventQueue.java:727) at java.desktop/java.awt.EventQueue$4.run(EventQueue.java:721) at java.base/java.security.AccessController.doPrivileged(Native Method) at java.base/java.security.ProtectionDomain$JavaSecurityAccessImpl.doIntersectionPrivilege(ProtectionDomain.java:85) at java.base/java.security.ProtectionDomain$JavaSecurityAccessImpl.doIntersectionPrivilege(ProtectionDomain.java:95) at java.desktop/java.awt.EventQueue$5.run(EventQueue.java:751) at java.desktop/java.awt.EventQueue$5.run(EventQueue.java:749) at java.base/java.security.AccessController.doPrivileged(Native Method) at java.base/java.security.ProtectionDomain$JavaSecurityAccessImpl.doIntersectionPrivilege(ProtectionDomain.java:85) at java.desktop/java.awt.EventQueue.dispatchEvent(EventQueue.java:748) at com.intellij.ide.IdeEventQueue.defaultDispatchEvent(IdeEventQueue.java:898) at com.intellij.ide.IdeEventQueue.dispatchMouseEvent(IdeEventQueue.java:820) at com.intellij.ide.IdeEventQueue._dispatchEvent(IdeEventQueue.java:743) at com.intellij.ide.IdeEventQueue.lambda$dispatchEvent$6(IdeEventQueue.java:439) at com.intellij.openapi.progress.impl.CoreProgressManager.computePrioritized(CoreProgressManager.java:803) at com.intellij.ide.IdeEventQueue.lambda$dispatchEvent$7(IdeEventQueue.java:438) at com.intellij.openapi.application.TransactionGuardImpl.performActivity(TransactionGuardImpl.java:119) at com.intellij.ide.IdeEventQueue.performActivity(IdeEventQueue.java:604) at com.intellij.ide.IdeEventQueue.lambda$dispatchEvent$8(IdeEventQueue.java:436) at com.intellij.openapi.application.impl.ApplicationImpl.runIntendedWriteActionOnCurrentThread(ApplicationImpl.java:873) at com.intellij.ide.IdeEventQueue.dispatchEvent(IdeEventQueue.java:484) at java.desktop/java.awt.EventDispatchThread.pumpOneEventForFilters(EventDispatchThread.java:207) at java.desktop/java.awt.EventDispatchThread.pumpEventsForFilter(EventDispatchThread.java:128) at java.desktop/java.awt.EventDispatchThread.pumpEventsForHierarchy(EventDispatchThread.java:117) at java.desktop/java.awt.EventDispatchThread.pumpEvents(EventDispatchThread.java:113) at java.desktop/java.awt.EventDispatchThread.pumpEvents(EventDispatchThread.java:105) at java.desktop/java.awt.EventDispatchThread.run(EventDispatchThread.java:92)
torch.nn.CrossEntropyLoss().ignore_index is crashing when importing transfomers library
I am using layoutlm github which require python 3.6, transformer 2.9.0. I created an conda env:
name: env_test
channels:
- defaults
- conda-forge
dependencies:
- python=3.6
- pip=20.3.3
- pytorch=1.4.0
- cudatoolkit=10.1
- pip:
- transformers==2.9.0
I have the following test.py code to reproduce the issue:
import sys
import torch
from torch.nn import CrossEntropyLoss
from transformers import (
BertConfig,
__version__
)
print (sys.version)
print(torch.__version__)
print(__version__)
CrossEntropyLoss().ignore_index
print("success!")
Importing transformers library results in segmentation fault (core dumped) a when calling CrossEntropyLoss().ignore_index:
$python test.py
3.6.12 |Anaconda, Inc.| (default, Sep 8 2020, 23:10:56)
[GCC 7.3.0]
1.4.0
2.9.0
Segmentation fault (core dumped)
I tried to investigate a bit but I don't really see from where the problem is coming from:
gdb python
GNU gdb (Ubuntu 8.1.1-0ubuntu1) 8.1.1
Copyright (C) 2018 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from python...done.
(gdb) r test.py
Starting program: /home/jupyter/.conda-env/env_test/bin/python test.py
warning: Error disabling address space randomization: Operation not permitted
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
3.6.12 |Anaconda, Inc.| (default, Sep 8 2020, 23:10:56)
[GCC 7.3.0]
1.4.0
2.9.0
Program received signal SIGSEGV, Segmentation fault.
0x00007f97000055fb in ?? ()
(gdb) where
#0 0x00007f97000055fb in ?? ()
#1 0x00007f97f4755729 in void pybind11::cpp_function::initialize<void (*&)(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&), void, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, pybind11::name, pybind11::scope, pybind11::sibling>(void (*&)(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&), void (*)(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&), pybind11::name const&, pybind11::scope const&, pybind11::sibling const&)::{lambda(pybind11::detail::function_call&)#3}::_FUN(pybind11::detail::function_call&) ()
from /home/jupyter/.conda-env/env_test/lib/python3.6/site-packages/torch/lib/libtorch_python.so
#2 0x00007f97f436bca6 in pybind11::cpp_function::dispatcher(_object*, _object*, _object*) () from /home/jupyter/.conda-env/env_test/lib/python3.6/site-packages/torch/lib/libtorch_python.so
#3 0x000055fbadd73a14 in _PyCFunction_FastCallDict () at /tmp/build/80754af9/python_1599604603603/work/Objects/methodobject.c:231
#4 0x000055fbaddfba5c in call_function () at /tmp/build/80754af9/python_1599604603603/work/Python/ceval.c:4851
#5 0x000055fbade1e25a in _PyEval_EvalFrameDefault () at /tmp/build/80754af9/python_1599604603603/work/Python/ceval.c:3335
#6 0x000055fbaddf5c1b in _PyFunction_FastCall (globals=<optimized out>, nargs=1, args=<optimized out>, co=<optimized out>) at /tmp/build/80754af9/python_1599604603603/work/Python/ceval.c:4933
#7 fast_function () at /tmp/build/80754af9/python_1599604603603/work/Python/ceval.c:4968
#8 0x000055fbaddfbb35 in call_function () at /tmp/build/80754af9/python_1599604603603/work/Python/ceval.c:4872
#9 0x000055fbade1e25a in _PyEval_EvalFrameDefault () at /tmp/build/80754af9/python_1599604603603/work/Python/ceval.c:3335
#10 0x000055fbaddf5166 in _PyEval_EvalCodeWithName () at /tmp/build/80754af9/python_1599604603603/work/Python/ceval.c:4166
#11 0x000055fbaddf5e51 in fast_function () at /tmp/build/80754af9/python_1599604603603/work/Python/ceval.c:4992
#12 0x000055fbaddfbb35 in call_function () at /tmp/build/80754af9/python_1599604603603/work/Python/ceval.c:4872
#13 0x000055fbade1e25a in _PyEval_EvalFrameDefault () at /tmp/build/80754af9/python_1599604603603/work/Python/ceval.c:3335
#14 0x000055fbaddf5166 in _PyEval_EvalCodeWithName () at /tmp/build/80754af9/python_1599604603603/work/Python/ceval.c:4166
#15 0x000055fbaddf5e51 in fast_function () at /tmp/build/80754af9/python_1599604603603/work/Python/ceval.c:4992
#16 0x000055fbaddfbb35 in call_function () at /tmp/build/80754af9/python_1599604603603/work/Python/ceval.c:4872
#17 0x000055fbade1e25a in _PyEval_EvalFrameDefault () at /tmp/build/80754af9/python_1599604603603/work/Python/ceval.c:3335
#18 0x000055fbaddf5166 in _PyEval_EvalCodeWithName () at /tmp/build/80754af9/python_1599604603603/work/Python/ceval.c:4166
#19 0x000055fbaddf632c in _PyFunction_FastCallDict () at /tmp/build/80754af9/python_1599604603603/work/Python/ceval.c:5084
#20 0x000055fbadd73ddf in _PyObject_FastCallDict () at /tmp/build/80754af9/python_1599604603603/work/Objects/abstract.c:2310
#21 0x000055fbadd78873 in _PyObject_Call_Prepend () at /tmp/build/80754af9/python_1599604603603/work/Objects/abstract.c:2373
#22 0x000055fbadd7381e in PyObject_Call () at /tmp/build/80754af9/python_1599604603603/work/Objects/abstract.c:2261
#23 0x000055fbaddcc88b in slot_tp_init () at /tmp/build/80754af9/python_1599604603603/work/Objects/typeobject.c:6420
#24 0x000055fbaddfbd97 in type_call () at /tmp/build/80754af9/python_1599604603603/work/Objects/typeobject.c:915
#25 0x000055fbadd73bfb in _PyObject_FastCallDict () at /tmp/build/80754af9/python_1599604603603/work/Objects/abstract.c:2331
#26 0x000055fbaddfbbae in call_function () at /tmp/build/80754af9/python_1599604603603/work/Python/ceval.c:4875
#27 0x000055fbade1e25a in _PyEval_EvalFrameDefault () at /tmp/build/80754af9/python_1599604603603/work/Python/ceval.c:3335
#28 0x000055fbaddf6969 in _PyEval_EvalCodeWithName (qualname=0x0, name=<optimized out>, closure=0x0, kwdefs=0x0, defcount=0, defs=0x0, kwstep=2, kwcount=<optimized out>, kwargs=0x0, kwnames=0x0, argcount=0, args=0x0,
locals=0x7f98035bf1f8, globals=0x7f98035bf1f8, _co=0x7f980357aae0) at /tmp/build/80754af9/python_1599604603603/work/Python/ceval.c:4166
#29 PyEval_EvalCodeEx () at /tmp/build/80754af9/python_1599604603603/work/Python/ceval.c:4187
#30 0x000055fbaddf770c in PyEval_EvalCode (co=co#entry=0x7f980357aae0, globals=globals#entry=0x7f98035bf1f8, locals=locals#entry=0x7f98035bf1f8) at /tmp/build/80754af9/python_1599604603603/work/Python/ceval.c:731
#31 0x000055fbade77574 in run_mod () at /tmp/build/80754af9/python_1599604603603/work/Python/pythonrun.c:1025
#32 0x000055fbade77971 in PyRun_FileExFlags () at /tmp/build/80754af9/python_1599604603603/work/Python/pythonrun.c:978
#33 0x000055fbade77b73 in PyRun_SimpleFileExFlags () at /tmp/build/80754af9/python_1599604603603/work/Python/pythonrun.c:419
#34 0x000055fbade77c7d in PyRun_AnyFileExFlags () at /tmp/build/80754af9/python_1599604603603/work/Python/pythonrun.c:81
#35 0x000055fbade7b663 in run_file (p_cf=0x7fff210dc16c, filename=0x55fbaefa6dc0 L"test.py", fp=0x55fbaefda800) at /tmp/build/80754af9/python_1599604603603/work/Modules/main.c:340
#36 Py_Main () at /tmp/build/80754af9/python_1599604603603/work/Modules/main.c:811
#37 0x000055fbadd4543e in main () at /tmp/build/80754af9/python_1599604603603/work/Programs/python.c:69
#38 0x00007f9803fd6bf7 in __libc_start_main (main=0x55fbadd45350 <main>, argc=2, argv=0x7fff210dc378, init=<optimized out>, fini=<optimized out>, rtld_fini=<optimized out>, stack_end=0x7fff210dc368) at ../csu/libc-start.c:310
#39 0x000055fbade24d0b in _start () at ../sysdeps/x86_64/elf/start.S:103
(gdb
I am the following list of packages:
_libgcc_mutex 0.1 main defaults
_pytorch_select 0.2 gpu_0 defaults
blas 1.0 mkl defaults
ca-certificates 2020.12.8 h06a4308_0 defaults
certifi 2020.12.5 py36h06a4308_0 defaults
cffi 1.14.4 py36h261ae71_0 defaults
chardet 4.0.0 pypi_0 pypi
click 7.1.2 pypi_0 pypi
cudatoolkit 10.1.243 h6bb024c_0 defaults
cudnn 7.6.5 cuda10.1_0 defaults
dataclasses 0.8 pypi_0 pypi
filelock 3.0.12 pypi_0 pypi
idna 2.10 pypi_0 pypi
intel-openmp 2020.2 254 defaults
joblib 1.0.0 pypi_0 pypi
ld_impl_linux-64 2.33.1 h53a641e_7 defaults
libedit 3.1.20191231 h14c3975_1 defaults
libffi 3.3 he6710b0_2 defaults
libgcc-ng 9.1.0 hdf63c60_0 defaults
libstdcxx-ng 9.1.0 hdf63c60_0 defaults
mkl 2020.2 256 defaults
mkl-service 2.3.0 py36he8ac12f_0 defaults
mkl_fft 1.2.0 py36h23d657b_0 defaults
mkl_random 1.1.1 py36h0573a6f_0 defaults
ncurses 6.2 he6710b0_1 defaults
ninja 1.10.2 py36hff7bd54_0 defaults
numpy 1.19.2 py36h54aff64_0 defaults
numpy-base 1.19.2 py36hfa32c7d_0 defaults
openssl 1.1.1i h27cfd23_0 defaults
pip 20.3.3 py36h06a4308_0 defaults
pycparser 2.20 py_2 defaults
python 3.6.12 hcff3b4d_2 defaults
pytorch 1.4.0 cuda101py36h02f0884_0 defaults
readline 8.0 h7b6447c_0 defaults
regex 2020.11.13 pypi_0 pypi
requests 2.25.1 pypi_0 pypi
sacremoses 0.0.43 pypi_0 pypi
sentencepiece 0.1.94 pypi_0 pypi
setuptools 51.0.0 py36h06a4308_2 defaults
six 1.15.0 py36h06a4308_0 defaults
sqlite 3.33.0 h62c20be_0 defaults
tk 8.6.10 hbc83047_0 defaults
tokenizers 0.7.0 pypi_0 pypi
tqdm 4.55.1 pypi_0 pypi
transformers 2.9.0 pypi_0 pypi
urllib3 1.26.2 pypi_0 pypi
wheel 0.36.2 pyhd3eb1b0_0 defaults
xz 5.2.5 h7b6447c_0 defaults
zlib 1.2.11 h7b6447c_3 defaults
What is responsible fo this core dump (I have a VM with 30 GB of memory) ? Seems to be related to transformers. Some dependency issues not catched by conda ? This piece of code seems to work with the latest version of transformers 4.1.1 but this is not compatible with layoutlm. Any suggestions?
It seems something was broken on layoutlm with pytorch 1.4 related issue. Switching to pytorch 1.6 fix the issue with the core dump, and the layoutlm code run without any modification.
anaconda env: cupti64_100.dll not found
# output metadata end of epochs
print('\nAdding run metadata for epoch ' + str(batch_idx) + '\n')
run_options = tf.RunOptions(trace_level=tf.RunOptions.FULL_TRACE)
run_metadata = tf.RunMetadata()
summary, _ = sess.run([merged, train_step],
feed_dict={x: batch_xs, y_: batch_ys, keep_prob: 1.0}, # test_xs, test,_ys
options=run_options,
run_metadata=run_metadata)
train_writer.add_run_metadata(run_metadata, 'step%03d' % batch_idx)
train_writer.add_summary(summary, batch_idx)
2019-11-10 16:07:00.007157: I tensorflow/core/profiler/lib/profiler_session.cc:174] Profiler session started.
2019-11-10 16:07:00.012641: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Could not dlopen library 'cupti64_100.dll'; dlerror: cupti64_100.dll not found
2019-11-10 16:07:00.018396: W tensorflow/core/profiler/lib/profiler_session.cc:182] Encountered error while starting profiler: Unavailable: CUPTI error: CUPTI could not be loaded or symbol could not be found.
2019-11-10 16:07:00.108662: I tensorflow/core/platform/default/device_tracer.cc:641] Collecting 0 kernel records, 0 memcpy records.
2019-11-10 16:07:00.114275: E tensorflow/core/platform/default/device_tracer.cc:68] CUPTI error: CUPTI could not be loaded or symbol could not be found.
W1110 16:07:00.226252 11440 deprecation_wrapper.py:119] From training.py:284: The name tf.train.Saver is deprecated. Please use tf.compat.v1.train.Saver instead.
my setup:
-Python 3.7.3
-cudatoolkit 10.0.130
-cudnn 7.6.0
-tensorflow 1.14.0
-tensorflow-gpu 1.14.0
i using Anaconda environment.
please help me what should i do?
Providing the solution here (Answer Section), even though it is present in the Comment Section, for the benefit of the community. Installed nvidia cuda 10.0.130 and copy cudnn 7.6.0 to C:\Program Files has resolved the issue.
Theano: GPU Usage on a New Jetson TX1
(Very long error messages in the question below. TL;DR, here is the specific question: Why does this test code not execute on the TX1's GPU, and what do I need to do to make it do so?)
I have just flashed and installed a brand new Nvidia Jetson TX1, with JetPack 2.3. I am trying to get Theano installed on the TX1 in such a way as to enable the use of the on-board GPU, for further machine learning and neural network applications.
However, I cannot seem to get the GPU itself to work.
The install of Theano was taken from here:
sudo apt-get install python-numpy python-scipy python-dev python-pip python-nose g++ libblas-dev git
pip install --upgrade --no-deps git+git://github.com/Theano/Theano.git --user # Need Theano 0.8(not yet released) or more recent
Theano version installed was 0.9.0.dev2, python is version 2.7.12.
I used the test script from here :
from theano import function, config, shared, tensor
import numpy
import time
vlen = 10 * 30 * 768 # 10 x #cores x # threads per core
iters = 1000
rng = numpy.random.RandomState(22)
x = shared(numpy.asarray(rng.rand(vlen), config.floatX))
f = function([], tensor.exp(x))
print(f.maker.fgraph.toposort())
t0 = time.time()
for i in range(iters):
r = f()
t1 = time.time()
print("Looping %d times took %f seconds" % (iters, t1 - t0))
print("Result is %s" % (r,))
if numpy.any([isinstance(x.op, tensor.Elemwise) and
('Gpu' not in type(x.op).__name__)
for x in f.maker.fgraph.toposort()]):
print('Used the cpu')
else:
print('Used the gpu')
When running as recommended:
THEANO_FLAGS=device=cuda0 python gpu_tutorial1.py
I get the following response, full of errors, warnings, and an execution on the CPU rather than the GPU:
ERROR (theano.gpuarray): pygpu was configured but could not be imported
Traceback (most recent call last):
File "/home/ubuntu/.local/lib/python2.7/site-packages/theano/gpuarray/__init__.py", line 21, in <module>
import pygpu
ImportError: No module named pygpu
WARNING (theano.gof.cmodule): OPTIMIZATION WARNING: Theano was not able to find the default g++ parameters. This is needed to tune the compilation to your specific CPU. This can slow down the execution of Theano functions. Please submit the following lines to Theano's mailing list so that we can fix this problem:
['# 1 "<stdin>"\n', '# 1 "<built-in>"\n', '# 1 "<command-line>"\n', '# 1 "/usr/include/stdc-predef.h" 1 3 4\n', '# 1 "<command-line>" 2\n', '# 1 "<stdin>"\n', 'Using built-in specs.\n', 'COLLECT_GCC=/usr/bin/g++\n', 'Target: aarch64-linux-gnu\n', "Configured with: ../src/configure -v --with-pkgversion='Ubuntu/Linaro 5.4.0-6ubuntu1~16.04.2' --with-bugurl=file:///usr/share/doc/gcc-5/README.Bugs --enable-languages=c,ada,c++,java,go,d,fortran,objc,obj-c++ --prefix=/usr --program-suffix=-5 --enable-shared --enable-linker-build-id --libexecdir=/usr/lib --without-included-gettext --enable-threads=posix --libdir=/usr/lib --enable-nls --with-sysroot=/ --enable-clocale=gnu --enable-libstdcxx-debug --enable-libstdcxx-time=yes --with-default-libstdcxx-abi=new --enable-gnu-unique-object --disable-libquadmath --enable-plugin --with-system-zlib --disable-browser-plugin --enable-java-awt=gtk --enable-gtk-cairo --with-java-home=/usr/lib/jvm/java-1.5.0-gcj-5-arm64/jre --enable-java-home --with-jvm-root-dir=/usr/lib/jvm/java-1.5.0-gcj-5-arm64 --with-jvm-jar-dir=/usr/lib/jvm-exports/java-1.5.0-gcj-5-arm64 --with-arch-directory=aarch64 --with-ecj-jar=/usr/share/java/eclipse-ecj.jar --enable-multiarch --enable-fix-cortex-a53-843419 --disable-werror --enable-checking=release --build=aarch64-linux-gnu --host=aarch64-linux-gnu --target=aarch64-linux-gnu\n", 'Thread model: posix\n', 'gcc version 5.4.0 20160609 (Ubuntu/Linaro 5.4.0-6ubuntu1~16.04.2) \n', "COLLECT_GCC_OPTIONS='-E' '-v' '-shared-libgcc' '-mlittle-endian' '-mabi=lp64'\n", ' /usr/lib/gcc/aarch64-linux-gnu/5/cc1 -E -quiet -v -imultiarch aarch64-linux-gnu - -mlittle-endian -mabi=lp64 -fstack-protector-strong -Wformat -Wformat-security\n', 'ignoring nonexistent directory "/usr/local/include/aarch64-linux-gnu"\n', 'ignoring nonexistent directory "/usr/lib/gcc/aarch64-linux-gnu/5/../../../../aarch64-linux-gnu/include"\n', '#include "..." search starts here:\n', '#include <...> search starts here:\n', ' /usr/lib/gcc/aarch64-linux-gnu/5/include\n', ' /usr/local/include\n', ' /usr/lib/gcc/aarch64-linux-gnu/5/include-fixed\n', ' /usr/include/aarch64-linux-gnu\n', ' /usr/include\n', 'End of search list.\n', 'COMPILER_PATH=/usr/lib/gcc/aarch64-linux-gnu/5/:/usr/lib/gcc/aarch64-linux-gnu/5/:/usr/lib/gcc/aarch64-linux-gnu/:/usr/lib/gcc/aarch64-linux-gnu/5/:/usr/lib/gcc/aarch64-linux-gnu/\n', 'LIBRARY_PATH=/usr/lib/gcc/aarch64-linux-gnu/5/:/usr/lib/gcc/aarch64-linux-gnu/5/../../../aarch64-linux-gnu/:/usr/lib/gcc/aarch64-linux-gnu/5/../../../../lib/:/lib/aarch64-linux-gnu/:/lib/../lib/:/usr/lib/aarch64-linux-gnu/:/usr/lib/../lib/:/usr/lib/gcc/aarch64-linux-gnu/5/../../../:/lib/:/usr/lib/\n', "COLLECT_GCC_OPTIONS='-E' '-v' '-shared-libgcc' '-mlittle-endian' '-mabi=lp64'\n"]
[Elemwise{exp,no_inplace}(<TensorType(float64, vector)>)]
Looping 1000 times took 12.736936 seconds
Result is [ 1.23178032 1.61879341 1.52278065 ..., 2.20771815 2.29967753
1.62323285]
Used the cpu
When I change the device flag to 'gpu':
THEANO_FLAGS=device=gpu python gpu_tutorial1.py
things improve somewhat, in that the NVIDIA Tegra X1 is at least found, although it is ultimately not used:
Using gpu device 0: NVIDIA Tegra X1 (CNMeM is disabled, cuDNN 5105)
WARNING (theano.gof.cmodule): OPTIMIZATION WARNING: Theano was not able to find the default g++ parameters. This is needed to tune the compilation to your specific CPU. This can slow down the execution of Theano functions. Please submit the following lines to Theano's mailing list so that we can fix this problem:
['# 1 "<stdin>"\n', '# 1 "<built-in>"\n', '# 1 "<command-line>"\n', '# 1 "/usr/include/stdc-predef.h" 1 3 4\n', '# 1 "<command-line>" 2\n', '# 1 "<stdin>"\n', 'Using built-in specs.\n', 'COLLECT_GCC=/usr/bin/g++\n', 'Target: aarch64-linux-gnu\n', "Configured with: ../src/configure -v --with-pkgversion='Ubuntu/Linaro 5.4.0-6ubuntu1~16.04.2' --with-bugurl=file:///usr/share/doc/gcc-5/README.Bugs --enable-languages=c,ada,c++,java,go,d,fortran,objc,obj-c++ --prefix=/usr --program-suffix=-5 --enable-shared --enable-linker-build-id --libexecdir=/usr/lib --without-included-gettext --enable-threads=posix --libdir=/usr/lib --enable-nls --with-sysroot=/ --enable-clocale=gnu --enable-libstdcxx-debug --enable-libstdcxx-time=yes --with-default-libstdcxx-abi=new --enable-gnu-unique-object --disable-libquadmath --enable-plugin --with-system-zlib --disable-browser-plugin --enable-java-awt=gtk --enable-gtk-cairo --with-java-home=/usr/lib/jvm/java-1.5.0-gcj-5-arm64/jre --enable-java-home --with-jvm-root-dir=/usr/lib/jvm/java-1.5.0-gcj-5-arm64 --with-jvm-jar-dir=/usr/lib/jvm-exports/java-1.5.0-gcj-5-arm64 --with-arch-directory=aarch64 --with-ecj-jar=/usr/share/java/eclipse-ecj.jar --enable-multiarch --enable-fix-cortex-a53-843419 --disable-werror --enable-checking=release --build=aarch64-linux-gnu --host=aarch64-linux-gnu --target=aarch64-linux-gnu\n", 'Thread model: posix\n', 'gcc version 5.4.0 20160609 (Ubuntu/Linaro 5.4.0-6ubuntu1~16.04.2) \n', "COLLECT_GCC_OPTIONS='-E' '-v' '-shared-libgcc' '-mlittle-endian' '-mabi=lp64'\n", ' /usr/lib/gcc/aarch64-linux-gnu/5/cc1 -E -quiet -v -imultiarch aarch64-linux-gnu - -mlittle-endian -mabi=lp64 -fstack-protector-strong -Wformat -Wformat-security\n', 'ignoring nonexistent directory "/usr/local/include/aarch64-linux-gnu"\n', 'ignoring nonexistent directory "/usr/lib/gcc/aarch64-linux-gnu/5/../../../../aarch64-linux-gnu/include"\n', '#include "..." search starts here:\n', '#include <...> search starts here:\n', ' /usr/lib/gcc/aarch64-linux-gnu/5/include\n', ' /usr/local/include\n', ' /usr/lib/gcc/aarch64-linux-gnu/5/include-fixed\n', ' /usr/include/aarch64-linux-gnu\n', ' /usr/include\n', 'End of search list.\n', 'COMPILER_PATH=/usr/lib/gcc/aarch64-linux-gnu/5/:/usr/lib/gcc/aarch64-linux-gnu/5/:/usr/lib/gcc/aarch64-linux-gnu/:/usr/lib/gcc/aarch64-linux-gnu/5/:/usr/lib/gcc/aarch64-linux-gnu/\n', 'LIBRARY_PATH=/usr/lib/gcc/aarch64-linux-gnu/5/:/usr/lib/gcc/aarch64-linux-gnu/5/../../../aarch64-linux-gnu/:/usr/lib/gcc/aarch64-linux-gnu/5/../../../../lib/:/lib/aarch64-linux-gnu/:/lib/../lib/:/usr/lib/aarch64-linux-gnu/:/usr/lib/../lib/:/usr/lib/gcc/aarch64-linux-gnu/5/../../../:/lib/:/usr/lib/\n', "COLLECT_GCC_OPTIONS='-E' '-v' '-shared-libgcc' '-mlittle-endian' '-mabi=lp64'\n"]
[Elemwise{exp,no_inplace}(<TensorType(float64, vector)>)]
Looping 1000 times took 12.820628 seconds
Result is [ 1.23178032 1.61879341 1.52278065 ..., 2.20771815 2.29967753
1.62323285]
Used the cpu
I do plan to send the warning lines to the Theano mailing list, but that warning seems unrelated to what is currently my main issue: Why does this test code not execute on the TX1's GPU, and what do I need to do to make it do so?
It turns out that the recommended CLI invocation on that site is not correct. The correct invocation is: THEANO_FLAGS='device=gpu,floatX=float32' python gpu_tutorial1.py This is sufficient to execute on the GPU with a satisfactory speedup (noticeable and reported in the output) and to get rid of that gob-smackingly long error warning. Putting both those flags in a .theanorc file is also sufficient, and simplifies the invocation.
yesod devel fail when starting development server
I was starting to read about Yesod and I tried to install it on my Linux Mint Maya 64-bit machine. The haskell platform was already installed, I just installed the yesod-platform package with cabal install yesod-platform and all worked very well. Next step was to call yesod init with respond to all the questions - the application name: test. The problems starts when I called yesod devel... here it is what I've got: Yesod devel server. Press ENTER to quit Resolving dependencies... Configuring test-0.0.0... Rebuilding application... Building test-0.0.0... Preprocessing library test-0.0.0... Registering test-0.0.0... Starting development server: runghc -package-confdist/package.conf.inplace devel.hs devel.hs:2:1: Failed to load interface for `Application' It is not a module in the current program, or in any known package. Exit code: ExitFailure 1 I'm a little bit lost with this issue - I made a lot of research related to it but I have no luck... Can anyone help with this problem? Here it is my ghc-pkg list: /var/lib/ghc/package.conf.d: Cabal-1.14.0 GLUT-2.1.2.1 HDBC-2.3.1.1 HDBC-odbc-2.2.3.0 HTTP-4000.2.2 HUnit-1.2.4.2 MonadCatchIO-mtl-0.3.0.4 OpenGL-2.2.3.1 QuickCheck-2.4.2 SHA-1.5.0.0 aeson-0.6.0.0 ansi-terminal-0.5.5 array-0.4.0.0 asn1-data-0.6.1.3 attoparsec-0.10.1.1 attoparsec-conduit-0.2.0 attoparsec-enumerator-0.3 authenticate-1.0.0.1 base-4.5.0.0 base-unicode-symbols-0.2.2.3 base64-bytestring-0.1.1.0 bin-package-db-0.0.0.0 binary-0.5.1.0 blaze-builder-0.3.1.0 blaze-builder-conduit-0.2.0 blaze-html-0.4.3.1 byteorder-1.0.3 bytestring-0.9.2.1 bytestring-lexing-0.4.0 case-insensitive-0.4.0.1 cereal-0.3.5.1 certificate-1.1.0 cgi-3001.1.8.2 clientsession-0.7.4.1 conduit-0.2.0 containers-0.4.2.1 convertible-1.0.11.0 cookie-0.4.0 cprng-aes-0.2.3 crypto-api-0.9 crypto-conduit-0.1.3.1 crypto-pubkey-types-0.1.0 cryptocipher-0.3.0 cryptohash-0.7.4 css-text-0.1.1 data-default-0.3.0 deepseq-1.3.0.0 directory-1.1.0.2 dlist-0.5 email-validate-0.2.8 entropy-0.2.1 enumerator-0.4.18 extensible-exceptions-0.1.1.4 failure-0.2.0 fast-logger-0.0.2 fgl-5.4.2.4 file-embed-0.0.4.1 filepath-1.3.0.0 (ghc-7.4.1) ghc-prim-0.2.0.0 hamlet-0.10.8 hashable-1.1.2.3 haskell-src-1.0.1.5 (haskell2010-1.1.0.1) (haskell98-2.0.0.1) hjsmin-0.0.15 hoopl-3.8.7.3 hpc-0.5.1.1 html-1.0.1.2 http-conduit-1.2.5 http-date-0.0.2 http-types-0.6.9 integer-gmp-0.4.0.0 language-javascript-0.4.10 largeword-1.0.1 lifted-base-0.1.0.3 mime-mail-0.4.1.1 monad-control-0.3.1 mtl-2.0.1.0 network-2.3.0.10 old-locale-1.0.0.4 old-time-1.1.0.0 parallel-3.2.0.2 parsec-3.1.2 path-pieces-0.1.0 persistent-0.8.0.1 persistent-template-0.8.1.1 pool-conduit-0.0.0.1 pretty-1.1.1.0 primitive-0.4.1 process-1.1.0.1 pureMD5-2.1.0.3 pwstore-fast-2.2 random-1.0.1.1 ranges-0.2.4 regex-base-0.93.2 regex-compat-0.95.1 regex-posix-0.95.1 resource-pool-0.2.1.0 rts-1.0 safe-0.3.3 semigroups-0.8 shakespeare-0.10.3.1 shakespeare-css-0.10.7 shakespeare-i18n-0.0.1 shakespeare-js-0.11.0.1 shakespeare-text-0.10.5 simple-sendfile-0.2.0 skein-0.1.0.5 socks-0.4.1 stm-2.2.0.1 syb-0.3.6 system-filepath-0.4.6 tagged-0.2.3.1 tagsoup-0.12.6 template-haskell-2.7.0.0 text-0.11.1.13 time-1.4 tls-0.9.0 tls-extra-0.4.3 transformers-0.2.2.0 transformers-base-0.4.1 unix-2.5.1.0 unix-compat-0.3.0.1 unordered-containers-0.1.4.6 utf8-light-0.4.0.1 utf8-string-0.3.7 vault-0.1.0.0 vector-0.9.1 wai-1.1.0.1 wai-app-static-1.1.2 wai-extra-1.1.0.1 wai-logger-0.1.4 warp-1.1.0.1 xhtml-3000.2.0.5 xml-conduit-0.5.3 xml-types-0.3.1 xss-sanitize-0.3.1 yaml-0.5.2 yesod-0.10.1.3 yesod-auth-0.8.1.1 yesod-core-0.10.2.1 yesod-form-0.4.2 yesod-json-0.3.1 yesod-persistent-0.3.1 yesod-routes-0.0.1 yesod-static-0.10.1 zlib-0.5.3.3 zlib-bindings-0.0.3.2 zlib-conduit-0.2.0 /home/ionica/.ghc/x86_64-linux-7.4.1/package.conf.d: Cabal-1.16.0.1 HDBC-2.3.1.1 HDBC-odbc-2.3.1.0 HDBC-sqlite3-2.3.3.0 HaXml-1.23.3 SHA-1.5.1 aeson-0.6.0.2 ansi-terminal-0.5.5 asn1-data-0.7.1 attoparsec-0.10.2.0 attoparsec-conduit-0.5.0.2 authenticate-1.3.1.1 base-unicode-symbols-0.2.2.4 base64-bytestring-1.0.0.0 blaze-builder-0.3.1.0 blaze-builder-conduit-0.5.0.1 blaze-html-0.5.1.0 blaze-markup-0.5.1.1 byteorder-1.0.3 cairo-0.12.3.1 case-insensitive-0.4.0.3 cereal-0.3.5.2 certificate-1.2.8 cipher-aes-0.1.2 clientsession-0.8.0.1 conduit-0.5.2.5 conduit-0.5.2.6 convertible-1.0.11.1 cookie-0.4.0.1 cpphs-1.14 cprng-aes-0.2.4 cpu-0.1.1 crypto-api-0.10.2 crypto-conduit-0.4.0.1 crypto-pubkey-types-0.1.1 cryptocipher-0.3.5 cryptohash-0.7.5 css-text-0.1.1 data-default-0.5.0 date-cache-0.3.0 derive-2.5.11 dlist-0.5 email-validate-0.2.8 entropy-0.2.1 failure-0.2.0.1 fast-logger-0.3.1 file-embed-0.0.4.6 filesystem-conduit-0.5.0.1 ghc-paths-0.1.0.8 gio-0.12.3 glib-0.12.3.1 gtk-0.12.3.1 hamlet-1.1.1 hashable-1.1.2.5 haskell-src-exts-1.13.5 hjsmin-0.1.3 hspec-1.3.0.1 hspec-expectations-0.3.0.2 html-conduit-0.1.0.2 http-conduit-1.6.1.2 http-date-0.0.2 http-types-0.7.3.0.1 language-javascript-0.5.5 largeword-1.0.3 lifted-base-0.1.2 lifted-base-0.2 mime-mail-0.4.1.2 mime-types-0.1.0.0 monad-control-0.3.1.4 monad-logger-0.2.1 network-conduit-0.6.0 network-conduit-0.6.1.1 pango-0.12.3 parallel-3.2.0.2 parallel-3.2.0.3 parallel-io-0.3.2.1 path-pieces-0.1.2 pem-0.1.1 persistent-1.0.1.3 persistent-sqlite-0.8.0 persistent-sqlite-1.0.0 persistent-template-1.0.0.2 polyparse-1.8 pool-conduit-0.1.0.3 primitive-0.5 primitive-0.5.0.1 pureMD5-2.1.2.1 pwstore-fast-2.3 ranges-0.2.4 resource-pool-0.2.1.1 resourcet-0.4.0.1 resourcet-0.4.0.2 safe-0.3.3 semigroups-0.8.4.1 shakespeare-1.0.1.4 shakespeare-css-1.0.1.5 shakespeare-i18n-1.0.0.2 shakespeare-js-1.0.0.6 shakespeare-text-1.0.0.5 silently-1.2.0.2 simple-sendfile-0.2.8 skein-0.1.0.9 socks-0.4.2 split-0.1.4.3 stringsearch-0.3.6.4 system-fileio-0.3.10 system-filepath-0.4.7 tagged-0.4.4 tagsoup-0.12.8 tagstream-conduit-0.5.3 tar-0.3.2.0 tar-0.4.0.1 tls-0.9.11 tls-extra-0.4.7 transformers-base-0.4.1 uniplate-1.6.7 unix-compat-0.4.0.0 unix-time-0.1.2 unordered-containers-0.2.2.1 utf8-string-0.3.7 vault-0.2.0.1 vector-0.10 vector-0.10.0.1 void-0.5.8 wai-1.3.0.1 wai-app-static-1.3.0.2 wai-extra-1.3.0.3 wai-logger-0.3.0 wai-test-1.3.0 warp-1.3.4.1 xml-conduit-1.0.3.1 xml-types-0.3.3 xss-sanitize-0.3.2 yaml-0.8.1 yesod-1.1.2 yesod-auth-1.1.1.2 yesod-core-1.1.3.1 yesod-default-0.6.1 yesod-default-1.1.0.2 yesod-form-1.1.3 yesod-json-1.1.0 yesod-persistent-1.1.0 yesod-platform-1.1.4 yesod-routes-1.1.0.1 yesod-routes-1.1.0.2 yesod-static-1.1.0.1 yesod-test-0.3.0.1 zlib-bindings-0.1.1.1 zlib-conduit-0.5.0.1
After a complete clean up of the system - I mean uninstall haskell-platform, yesod-platform and delete files related to haskell, yesod and ghc - then a fresh install for haskell-platform + yesod-platform all goes splendid.