How can i download wiki part in one txt file - text

I need a huge natural text file for machine learning and Wikipedia dump is great for this purpose. So how can i download several gb of text in some language (non-eng) without xml tags (just content)?

You could grab a dump of all content of a Wikipedia of your choice from dumps.wikimedia.org. You will likely want one of the *wiki-20160501-pages-articles.xml files. Then, you could strip all XML tags from the dump using a tool like xmlstarlet:
xml sel -t -c "//text()" fywiki-20160501-pages-articles.xml > articles.txt
However, the text in a Wikipedia dump will be wiki markup, not natural text. You strip everything that's not alphanumeric with something like sed:
cat dump.txt | sed 's/\W/ /g'
This doesn't give you a clean corpus (for example, wikimarkup keywords and html entities will still be in your dump file) but it may be close enough for your purpose.

Phase a:
Go to dumps.wikimedia.org. Find a dump that fits your request. For machine learning - best way is to download "All pages, current versions only" dump of your language. Download and unzip it.
Phase b:
As the dump is xml file with a wiki-markup syntax of content - it has to be converted to plain text. The best solution i've found is to use this toolkit - https://github.com/yohasebe/wp2txt . It needs no much memory and works well.
Phase c:
wp2txt produces a hundreds of 10mb txt file, so we need to concatenate them. Use
cat * > all.txt
at nix systems or
cat dump.txt | sed 's/\W/ /g'
for windows one.
p.s. Also i've found better semi-legal solution for ML case. The solution is to download some huge txt-literature library. Have a nice learning!

for Python try this after downloading .xml dump
pip install wiki-dump-reader
https://pypi.org/project/wiki-dump-reader/

Related

How to convert markdown to pdf in command line

I need to convert the GitHub README.md file to pdf. I tried many modules, those are not working fine. Is there any new tool to get the exact pdf format. In this website is providing good conversion format of pdf. http://www.markdowntopdf.com/
I need command line tool like this format.
Try this software:
https://github.com/BlueHatbRit/mdpdf
Or explain what tools you've tried and why those are not working fine.
Also check this question on superuser:
https://superuser.com/questions/689056/how-can-i-convert-github-flavored-markdown-to-a-pdf
Pandoc
I've personally liked using pandoc as it support a wide range of input and output formats.
Installation
Pandoc is available in most repositories: sudo apt install pandoc
Usage
Sometimes, pandoc can tell the formats to use which makes converting easy. However, I find that this often interprets the input format as plain text which might not be what you want:
pandoc README.md -o README.pdf
Instead, you might want to be explicit about the input/output formats to ensure a better conversion. In the below case, I'm specifically claiming the README.md is in Github-Flavored Markdown:
pandoc --from=gfm --to=pdf -o README.pdf README.md
Again, there are quite a few different formats and options to choose from but to be honest, the basics suffice for the majority of my needs.
I found md-to-pdf very useful.
Examples:
– Convert ./file.md and save to ./file.pdf
$ md-to-pdf file.md
– Convert all markdown files in current directory
$ md-to-pdf ./*.md
– Convert all markdown files in current directory recursively
$ md-to-pdf ./**/*.md
– Convert and enable watch mode
$ md-to-pdf ./*.md -w
And many more options.

How can doc/docx files be converted to markdown or structured text?

Is there a program or workflow to convert .doc or .docx files to Markdown or similar text?
PS: Ideally, I would welcome the option that a specific font (e.g. consolas) in the MS Word document will be rendered to text-code: ```....```.
Pandoc supports conversion from docx to markdown directly:
pandoc -f docx -t markdown foo.docx -o foo.markdown
Several markdown formats are supported:
-t gfm (GitHub-Flavored Markdown)
-t markdown_mmd (MultiMarkdown)
-t markdown (pandoc’s extended Markdown)
-t markdown_strict (original unextended Markdown)
-t markdown_phpextra (PHP Markdown Extra)
-t commonmark (CommonMark Markdown)
docx -> markdown
Specifically regarding the question (docx --> markdown), use the Writeage plugin for Microsoft Word. It also works the other way round markdown --> docx.
More Options
Use a Conversion Tool for multi-file conversion.
Use a WYSIWYG Editor for single files and superior fonts.
Which Conversion Tools?
I've tested these three: (1) Pandoc (2) Mammoth (3) w2m
Pandoc
By far the superior tool for conversions with support for a multitude of file types (see Pandoc's man page for supported file types):
pandoc -f docx -t gfm somedoc.docx -o somedoc.md
NB
To get pandoc to export markdown tables ('pipe_tables' in pandoc) use multimarkdown or gfm output formats.
If formatting to PDF, pandoc uses LaTeX templates for this so you may need to install the LaTeX package for your OS if that command does not work out of the box. Instructions at LaTeX Installation
Which WYSIWYG Editors?
For docx, use Writeage.
Maintaining Superior Fonts
If you wish to preserve unicode characters, emojis and maintain superior fonts, you'll get some milage from the editors below when using copy-and-paste operations between file formats. Note, these do not read or write natively to docx.
Typora
iaWriter
Markdown Viewer for Chrome.
Programatic Equivalent
For a programatic equivalent, you might get some results by calling a different pdf-engine and their respective options but I haven't tested this. The pandoc defaults to 'pdflatex'.
pandoc --pdf-engine=
pandoc --pdf-engine-opt=STRING
Update: A4 vs US Letter
For outside the US, set the geometry variable:
pandoc -s -V geometry:a4paper -o outfile.pdf infile.md
Footnote
Its worth mentioning here - what's not obvious when discovering Markdown is that MultiMarkdown is by far the most feature rich markdown format.
MultiMarkdown supports amongst other things - metadata, table of contents, footnotes, maths, tables and YAML.
But Github's default format uses gfm which also supports tables. I use gfm for Github/GitLab and MultiMarkdown for everything else.
Given that you asked this question on stackoverflow you're probably wanting a programmatic or command line solution for which I've included another answer.
However, an alternative solution might be to use the Writage Markdown plugin for Microsoft Word.
Writage turns Word into your Markdown WYSIWYG editor, so you will be able to open a Markdown file and edit it like you normally edit any document in Microsoft Word. Also it will be possible to save your Word document as a Markdown file without any other converters.
Under the covers, Writage uses Pandoc that you'll also need to install for this plugin to work.
It currently supports the following Markdown elements:
Headings
Lists (numbered and bulleted)
Links
Font styles such as bold, italic
Tables
Footnotes
This might be the ideal solution for many end users as they won't need to install or run any command line tools - but rather just stick with what they are most familiar.
Mammoth is best known as a Word to HTML converter but it now supports a Markdown writer module. When I last checked, Mammoth Markdown support was still in its early stages, so you may find some features are unsupported. As usual ... check the website for the latest details.
Install
To use the Javascript version ... install NodeJS and then install Mammoth:
npm install -g mammoth
Command line
Command line to convert a Word document to Markdown ...
mammoth document.docx --output-format=markdown
API
NodeJS API to convert to Markdown ...
var mammoth = require("mammoth");
mammoth.convertToMarkdown({path: "path/to/document.docx"});
Features:
Mammoth Markdown writer currently supports:
Lists (numbered and bulleted)
Links
Font styles such as bold, italic
Images
The Mammoth command line tools and API have been ported to several languages:
With NO Markdown (May 2016):
.NET
Java/JVM
Wordpress
With Markdown:
Javascript
Python
You can use Word to Markdown (Ruby Gem) to convert it in one step. Conversion can be as simple as:
$ gem install word-to-markdown
$ w2m path/to/document.docx
It routes the document through LibreOffice, but also does it best to semantice headings based on their relative font size.
There's also a hosted version which would be as simple as drag-and-drop to convert.
Word to Markdown might be worth a shot, or the procedure described here using Calibre and Pandoc via HTMLZ, here's a bash script they use:
#!/bin/bash
mkdir temp
cp $1 temp
cd temp
ebook-convert $1 output.htmlz
unzip output.htmlz
cd ..
pandoc -f html -t markdown -o output.md temp/index.html
rm -R temp
From here:
unoconv -f html test.docx
pandoc -f html -t markdown -o test.md test.html
You can convert Word documents from within MS Word to Markdown using this Visual Basic Script:
https://gist.github.com/hawkrives/2305254
Follow the instructions under "To use the code" to create a new Macro in Word.
Note: This converts the currently open Word document ato Markdown, which removes all the Word formatting (headings, lists, etc.). First save the Word document you plan to converts, and then save the document again as a new document before running the macro. This way you can always go back to the original Word document to make changes.
There are more examples of Word to markdown VB scripts here:
https://www.mediawiki.org/wiki/Microsoft_Word_Macros
Here's an open-source web application built in Ruby to do this exact thing:
https://word2md.com
If you're using Linux, try Pandoc (first convert .doc/.docx into html with LibreOffice or something and then run it).
On Windows (or if Pandoc doesn't work), you can try this website (online demo, you can download it): Markdownify
For bulleted lists you can paste a list into Sublime Text and use multiselect ( tested ) or find and replace ( not tested ) to replace eg the proprietary MS Word characters with -, -- etc
This doesn't work with headings but it may be possible to use a similar technique with other elements.
For .doc Word files:
antiword -f some_file.doc
antiword's homepage: http://www.winfield.demon.nl/

Linux untar command shows file names as question marks

A while ago I had compressed an application using Linux "tar -cf" command. At that time some of the file names were in a different language.
Now when I uncompress using "tar -xf" it shows the file names in the other language as question marks.
Is there a way that when I uncompress it keeps the original file names as they were?
Your help is highly appreciated.
Good question ! It's expected that like any Unix command, tar could pipe its output to another program, if possible including filename data. A quick googling reveals that this is the case: as described in this blog post, GNU tar supports the --to-command parameter to write the output to a pipe, instead of directly operating on the directory.
http://osmanov-dev-notes.blogspot.com.br/2010/07/how-to-handle-filename-encodings-in.html
So it's a matter of writing a script to convert the filename to UTF-8, like it's done in the cited post. Another option, also described in the text, that becomes obvious after you read it is to simply extract everything and then write a script to convert every file in the directory. There's a trivial php script in the link that does this.
Finally, you can always write your own custom tar version with the help of scripting languages, and that's easy. Python, for example has the tarfile module built in the standard library:
http://docs.python.org/2/library/tarfile.html#examples
You could use TarFile.extractfile(), shutils.copyfileobj() and str.decode() in a loop to manually extract the files while changing the file name encoding.
References:
http://www.gnu.org/software/tar/manual/tar.html#SEC84
http://docs.python.org/2/library/tarfile.html
http://www.manpagez.com/man/1/iconv/

One-way diff file

I would like to generate diffs for the sake of doing incremental backups of an sql database.
Using the standard unix 'diff' tool generates unnecessarily large files, since they include the full text of deleted lines. I only need support to be able to patch in one direction (to generate the current db dump from the full dump and an incremental patch).
How would I go about doing this? I have tried so far using diff -e and patch -e, but it doesn't seem to be working correctly, as the resulting file is corrupt (possibly an issue with the 'ed' tool used in cygwin)
back in the old days, before Vim, there used to be a line-oriented UNIX editor called 'ed' ..
diff has an option built in ( -e option ) , with which you can create an edit script from the diff.
Check here: and look for the section "Edit Script"
http://en.wikipedia.org/wiki/Diff
http://docs.freebsd.org/info/diff/diff.info.ed_Scripts.html
here's an example:
http://www.araxis.com/merge/topic_diff_report_editscript.html
another way to do this is to create a patch file (see 'man patch')

Unable to search pdf-files' contents in terminal

I have pdf -files which contents I have not managed to search by any terminal program.
I can only search them by Acrobat Reader and Skim.
How can you search contents of pdf -files in terminal?
It seems that a better question is
How is the search done in the pdf viewers such as Acrobat Reader and Skim?
Perhaps, I need to make such a search tool if no such tools exist.
Try installing xpdf from MacPorts; it is supposed to come with a tool called pdftotext which should then allow you to search using grep.
pdftotext is indeed an excellent tool, but it produces very long lines; in order to grep you will want to break them up, e.g.,
pdftotext drscheme.pdf - | fmt | grep -i spidey
PDF files are usually compressed. PDF viewers such as Acrobat Reader and Skim search the contents by decompressing the PDF text into memory, and then searching that text. If you want to search from the command line, one possible suggestion is to use pdftk to decompress the PDF, and then use grep (or your favorite command line text searching utility) to find the desired text. For example:
# Search for the text "text_to_search_for", and print out 3 lines of context
# above and below each match
pdftk mydoc.pdf output - uncompress | grep -C3 text_to_search_for

Resources