I am concerned whether torch.solve() examine the condition of the coefficient matrix for a linear system and employ desirable preconditionings; thus I am curious about its implementation details. I have read through several answers trying to track down the source file but in vain. I hope somebody can help me to locate its definition in the ATen library.
I think it just uses LAPACK for CPU and CUBLAS for GPU, since torch.solve is listed under "BLAS and LAPACK Operations" on the official docs.
Then we're looking for wrapper code, which I believe is this part.
Related
I am trying to do research on batch normalization, and had to make some modifications for the pytorch BN code. I dig into the pytorch code and got stuck with torch.nn.functional.batch_norm, which references torch.batch_norm.
The problem is that torch.batch_norm cannot be further found in the torch library. Is there any way I can find the source code of this built-in function and re-implement it? Thanks!
It's there, but it's not defined in Python. They're defined in C++ in the aten/ directories.
For CPU, the implementation (one of them, it depends on whether or not the input is contiguous) is here: https://github.com/pytorch/pytorch/blob/420b37f3c67950ed93cd8aa7a12e673fcfc5567b/aten/src/ATen/native/Normalization.cpp#L61-L126
For CUDA, the implementation is here: https://github.com/pytorch/pytorch/blob/7aae51cdedcbf0df5a7a8bf50a947237ac4b3ee8/aten/src/ATen/native/cudnn/BatchNorm.cpp#L52-L143
I wanted to see how the conv1d module is implemented
https://pytorch.org/docs/stable/_modules/torch/nn/modules/conv.html#Conv1d. So I looked at functional.py but still couldn’t find the looping and cross-correlation computation.
Then I searched Github by keyword ‘conv1d’, checked conv.cpp https://github.com/pytorch/pytorch/blob/eb5d28ecefb9d78d4fff5fac099e70e5eb3fbe2e/torch/csrc/api/src/nn/modules/conv.cpp 1 but still couldn’t locate where the computation is happening.
My question is two-fold.
Where is the source code that "conv1d” is implemented?
In general, if I want to check how the modules are implemented, where is the best place to find? Any pointer to the documentation will be appreciated. Thank you.
It depends on the backend (GPU, CPU, distributed etc) but in the most interesting case of GPU it's pulled from cuDNN which is released in binary format and thus you can't inspect its source code. It's a similar story for CPU MKLDNN. I am not aware of any place where PyTorch would "handroll" it's own convolution kernels, but I may be wrong. EDIT: indeed, I was wrong as pointed out in an answer below.
It's difficult without knowing how PyTorch is structured. A lot of code is actually being autogenerated based on various markup files, as explained here. Figuring this out requires a lot of jumping around. For instance, the conv.cpp file you're linking uses torch::conv1d, which is defined here and uses at::convolution which in turn uses at::_convolution, which dispatches to multiple variants, for instance at::cudnn_convolution. at::cudnn_convolution is, I believe, created here via a markup file and just plugs in directly to cuDNN implementation (though I cannot pinpoint the exact point in code when that happens).
Below is an answer that I got from pytorch discussion board:
I believe the “handroll”-ed convolution is defined here: https://github.com/pytorch/pytorch/blob/master/aten/src/THNN/generic/SpatialConvolutionMM.c 3
The NN module implementations are here: https://github.com/pytorch/pytorch/tree/master/aten/src
The GPU version is in THCUNN and the CPU version in THNN
I have been asked to integrate a custom JPEG encoder kernel module to the linux tree. The description is too generic. Can anyone suggest where in kernel tree should this go? I mean under what category in the drivers? I am assuming this is going to be compiled as a module and not statically linked to the kernel. If I generalize the question where should any custom kernel module live in the kernel tree? Assume the kernel module is a video/audio decoder/encoder. In this case it is a JPEG encoder as I said.
Any help will be highly appreciated.
Thanks.
When I posted this question I did not have clarity as how drivers are categorized and placed in the kernel tree. So explored and this is what I found so far:
If I am integrating/writing a new driver e.g. Ring Oscillator (this device simply generates some frequencies given a input period value, the frequency number is fed to a random number generator). To my understanding this should go under linux/drivers/misc/ whereas someone argued this should go under linux/drivers/misc/. But apart from that there seems to no strict rule where this kind of drivers should go. So it is quite up to your discretion and judgment where you ultimately place it. I have given the details of the steps involved here.
I also had to integrate a jpeg encoder and I was confused where this driver should go. I initially thought I will place it under linux/drivers/media/ as suggested in the comments. But this turned out to be a matter of preference. Finally I integrated it as a new buildroot package. In case you are interested I have described it here.
This is my understanding so far. If anyone thinks if I have missed anything please kindly point out.
I had a brief exposure to CP and MiniZinc but I'm no expert.
I have a CP model, which I cannot post here ATM, implemented in MiniZinc.
I need to generate all the feasible solution for the problem. We expect to have just a "few" ones, say less than 1000, more than 100.
I tried to solve the model with the -a flag passed to minizinc ver. 1.6 but I notice a lot of solutions being printed are identical.
Here they refer to "projection". In another paper I read they used some "backtracking mechanism".
It's still not clear to me.
My questions then are:
what is the best way to generate only unique solutions from a CP model?
Is there a standard mechanism implemented in CP libraries like SCIP or Gecode? Does it have a common name?
Is it computationally efficient?
does minizinc support that? How do I access that feature?
Normally, CP systems give you just distinct solutions. I suspect that you have decision variables that are not printed (not in the output section) and you don't see that if these values are included in the solution it would be unique solutions.
In the question you linked to (this recent discussion), it is mentioned that Gecode's FlatZinc solver (at least the SVN version) now generates distinct solutions given a subset of decision variables in the output section. Other FlatZinc solver don't seem to have this feature.
If that doesn't answer your questions, please give more details of the model and example of the output (including the output section).
I've got some unused functionality in my codebase, but it's hard to identify. The code has evolved over the last year as I explore its problem space and possible solutions. What I'm needing to do is find that unused code so I can get rid of it. I'm happy if it deals with the problem on an exportable name basis.GHC has warnings that deal with non-exported unused code. Any tools specific to this task would be of interest.
However, I'm curious about a comprehensive cross referencing tool. I can find the unused code with such a tool. Years ago when I was working in C and assembler, I found that a good xref was a pretty handy tool, useful for many different purposes.
I'm getting nowhere with googling. Apparently in Haskell the dominant meaning of cross-reference is within literate programming. Though maybe something there would be useful.
I don’t know of such a tool, so in the past I have done a bit of a hack instead.
If you have a comprehensive test suite, you can run it with GHC’s code coverage tracing enabled. Compile with -fhpc and use hpc markup to generate annotated source. This gives you the union of unused code and untested code, both of which you would probably like to address anyway.
SourceGraph can give you a bunch of information which you may also find useful.
There is now a tool for this very purpose: https://hackage.haskell.org/package/weeder
It's been around since 2017, and while it has limitations, it definitely helps with large codebases.