How is the init process started in the Linux kernel? - linux

I am trying to understand the init process in the linux kernel which is the first process and is statically initialized with the INIT_TASK macro.
161 #define INIT_TASK(tsk) \
162 { \
163 .state = 0, \
164 .stack = &init_thread_info, \
165 .usage = ATOMIC_INIT(2), \
166 .flags = PF_KTHREAD, \
167 .prio = MAX_PRIO-20, \
168 .static_prio = MAX_PRIO-20, \
169 .normal_prio = MAX_PRIO-20, \
170 .policy = SCHED_NORMAL, \
171 .cpus_allowed = CPU_MASK_ALL, \
172 .nr_cpus_allowed= NR_CPUS, \
173 .mm = NULL, \
174 .active_mm = &init_mm, \
175 .se = { \
176 .group_node = LIST_HEAD_INIT(tsk.se.group_node), \
177 }, \
178 .rt = { \
179 .run_list = LIST_HEAD_INIT(tsk.rt.run_list), \
180 .time_slice = RR_TIMESLICE, \
181 }, \
182 .tasks = LIST_HEAD_INIT(tsk.tasks), \
183 INIT_PUSHABLE_TASKS(tsk) \
184 INIT_CGROUP_SCHED(tsk) \
185 .ptraced = LIST_HEAD_INIT(tsk.ptraced), \
186 .ptrace_entry = LIST_HEAD_INIT(tsk.ptrace_entry), \
187 .real_parent = &tsk, \
188 .parent = &tsk, \
189 .children = LIST_HEAD_INIT(tsk.children), \
190 .sibling = LIST_HEAD_INIT(tsk.sibling), \
191 .group_leader = &tsk, \
192 RCU_POINTER_INITIALIZER(real_cred, &init_cred), \
193 RCU_POINTER_INITIALIZER(cred, &init_cred), \
194 .comm = INIT_TASK_COMM, \
195 .thread = INIT_THREAD, \
196 .fs = &init_fs, \
197 .files = &init_files, \
198 .signal = &init_signals, \
199 .sighand = &init_sighand, \
200 .nsproxy = &init_nsproxy, \
201 .pending = { \
202 .list = LIST_HEAD_INIT(tsk.pending.list), \
203 .signal = {{0}}}, \
204 .blocked = {{0}}, \
205 .alloc_lock = __SPIN_LOCK_UNLOCKED(tsk.alloc_lock), \
206 .journal_info = NULL, \
207 .cpu_timers = INIT_CPU_TIMERS(tsk.cpu_timers), \
208 .pi_lock = __RAW_SPIN_LOCK_UNLOCKED(tsk.pi_lock), \
209 .timer_slack_ns = 50000, /* 50 usec default slack */ \
210 .pids = { \
211 [PIDTYPE_PID] = INIT_PID_LINK(PIDTYPE_PID), \
212 [PIDTYPE_PGID] = INIT_PID_LINK(PIDTYPE_PGID), \
213 [PIDTYPE_SID] = INIT_PID_LINK(PIDTYPE_SID), \
214 }, \
215 .thread_group = LIST_HEAD_INIT(tsk.thread_group), \
216 INIT_IDS \
217 INIT_PERF_EVENTS(tsk) \
218 INIT_TRACE_IRQFLAGS \
219 INIT_LOCKDEP \
220 INIT_FTRACE_GRAPH \
221 INIT_TRACE_RECURSION \
222 INIT_TASK_RCU_PREEMPT(tsk) \
223 INIT_CPUSET_SEQ \
224 INIT_VTIME(tsk) \
225 }
But I am not able to figure out
how it will be executed?
Where it is scheduled and
which lines of code in the linux kernel start executing immediately when we say we have scheduled this init_task task? Is there any function which it calls?

The kernel calls "init" as one of the very last things it does during kernel initialization. The function kernel_init() in init/main.c has the logic.
You will notice that the kernel tries four different combinations of init, and expects one of them to succeed. You will also notice that you can override what the kernel executes on startup by feeding the kernel command line parameter "init". So, you can say, for example, init=/bin/mystartup on the kernel command line and start your own custom application instead of the default /sbin/init. Notice also that on most modern systems, even embedded systems, /sbin/init is a soft link that points to the real executable.
To more generally answer your question, study this source file (main.c) you can see virtually all of the details of Linux kernel initialization, after the low-level assembly stuff and platform initialization, which, beyond the educational value, you shouldn't have to touch nor care about much.
The main mechanism is to call do_execve() with fixed arguments of argv_init and envp_init. The elf file is parsed and initial program counter (PC) is set as per the file. All memory management (mm) pages are mapped to the disks backing store. The code is set to run. On the initial PC fetch when it is scheduled, a page fault is generated which reads the first code page into memory. This is the same as any other execve() call.

Related

why run "python run_squad.py" doesn't work?

I want fine tune on squad with huggingface run_squad.py, but meet the following question:
1, when I use "--do_train" without "True" as following code, after 20 minutes runing,there is no models in output_dir:
!python run_squad.py \
--model_type bert \
--model_name_or_path bert-base-uncased \
--output_dir models/bert/ \
--data_dir data/squad \
--overwrite_output_dir \
--overwrite_cache \
--do_train \
--train_file train-v2.0.json \
--version_2_with_negative \
--do_lower_case \
--do_eval \
--predict_file dev-v2.0.json \
--per_gpu_train_batch_size 2 \
--learning_rate 3e-5 \
--num_train_epochs 2.0 \
--max_seq_length 384 \
--doc_stride 128 \
--threads 10 \
--save_steps 5000
2, when I use "--do_train=True" as following code, the error message is "run_squad.py: error: argument --do_train: ignored explicit argument 'True'":
!python run_squad.py \
--model_type bert \
--model_name_or_path bert-base-uncased \
--output_dir models/bert/ \
--data_dir data/squad \
--overwrite_output_dir \
--overwrite_cache \
--do_train=True \
--train_file train-v2.0.json \
--version_2_with_negative \
--do_lower_case \
--do_eval \
--predict_file dev-v2.0.json \
--per_gpu_train_batch_size 2 \
--learning_rate 3e-5 \
--num_train_epochs 2.0 \
--max_seq_length 384 \
--doc_stride 128 \
--threads 10 \
--save_steps 5000
3, when I use "--do_train True" as following code, the error message is "run_squad.py: error: unrecognized arguments: True":
!python run_squad.py \
--model_type bert \
--model_name_or_path bert-base-uncased \
--output_dir models/bert/ \
--data_dir data/squad \
--overwrite_output_dir \
--overwrite_cache \
--do_train True \
--train_file train-v2.0.json \
--version_2_with_negative \
--do_lower_case \
--do_eval \
--predict_file dev-v2.0.json \
--per_gpu_train_batch_size 2 \
--learning_rate 3e-5 \
--num_train_epochs 2.0 \
--max_seq_length 384 \
--doc_stride 128 \
--threads 10 \
--save_steps 5000
I run code in colab with GPU: Tesla P100-PCIE-16GB
Judging by the running time, I think the code didn't through training process, but I don't know how to set parameters in order to let training go.what should I do?

How to create a static Qt application with protobuf files on Linux?

I want to build a static qt application. First of all I build qt static version on my linux machine. In the first time when I builded static application, I received errors because my qt can't link my application with some linux *.so librarys. I fixed it by using
unix{ QMAKE_LFLAGS += -static-libgcc -static-libstdc++ }
and my application now runs on every linux platform, intependetly from local linux libraris, it doesn't linking with local linux .so files. I want to do that think also with my protobuf files. I didn't found in internet how to do that. So I'm asking for help in here. I add pro file below.
QT += core gui
greaterThan(QT_MAJOR_VERSION, 4): QT += widgets
TARGET = EthernetAntCtrl_app
TEMPLATE = app
ROOT = ./..
DESTDIR = $$ROOT/bin
TMP_PRG_DIR = $$ROOT/tmp
MOC_DIR = $$TMP_PRG_DIR/$$TARGET/moc
OBJECTS_DIR = $$TMP_PRG_DIR/$$TARGET/obj
UI_DIR = $$TMP_PRG_DIR/$$TARGET/ui
RCC_DIR = $$TMP_PRG_DIR/$$TARGET/resources
PROTOBUF_DIR = /usr/local/include/google/protobuf
include($$ROOT/status_monitoring/status_monitoring.pri)
include($$ROOT/qt-propertybrowser/src/qtpropertybrowser.pri)
DEFINES += QT_DEPRECATED_WARNINGS
#-static-libprotobuf
unix{
QMAKE_LFLAGS += -static-libgcc -static-libstdc++
}
SOURCES += \
main.cpp \
mainwindow.cpp \
interconnect.cpp \
switch.cpp \
../Common/MyLog.cpp \
../DevMan/DevmanLibraryController/CmdBase.cpp \
../DevMan/DevmanLibraryController/DevmanLibraryController.cpp \
../DevMan/Common/Names/CommonNames.cpp \
../DevMan/Common/Names/SystemMessageNames.cpp \
../DevMan/DevmanLibraryController/ConnectMsgs/TcpConnectMessage.cpp \
../DevMan/DevmanLibraryController/ConnectMsgs/TcpDisconnectMessage.cpp \
../DevMan/DevmanLibraryController/InitDeviceManagerMsgs/ResultMessage.cpp \
Msgs/InitDeviceManager_AntCtrl.cpp \
textedit.cpp \
pushbutton.cpp \
controller.cpp \
Msgs/CustomNames.cpp \
Msgs/AntCntrlMessage.cpp \
button_style.cpp \
ClientParameterManager.cpp \
../DevMan/DevmanLibraryController/ConnectMsgs/UdpConnectMessage.cpp \
../DevMan/DevmanLibraryController/ConnectMsgs/UdpDisconnectMessage.cpp \
LedButton.cpp
HEADERS += \
switch.h \
mainwindow.h \
interconnect.h \
structureofpackets.h \
../Common/MacroDefines.h \
../Common/MyLog.h \
../DevMan/DevmanLibraryController/CmdBase.h \
../DevMan/DevmanLibraryController/DevmanLibraryController.h \
../DevMan/Common/Names/CommonNames.h \
../DevMan/Common/Names/SystemMessageNames.h \
../DevMan/DevmanLibraryController/ConnectMsgs/TcpConnectMessage.h \
../DevMan/DevmanLibraryController/ConnectMsgs/TcpDisconnectMessage.h \
../DevMan/DevmanLibraryController/InitDeviceManagerMsgs/ResultMessage.h \
Msgs/InitDeviceManager_AntCtrl.h \
textedit.h \
pushbutton.h \
controller.h \
Msgs/CustomNames.h \
Msgs/AntCntrlMessage.h \
button_style.h \
ClientParameterManager.h \
../DevMan/DevmanLibraryController/ConnectMsgs/UdpConnectMessage.h \
../DevMan/DevmanLibraryController/ConnectMsgs/UdpDisconnectMessage.h \
LedButton.h
FORMS += \
mainwindow.ui
defineTest(copyToDestdir) {
files = $$1
#copyToDestdir($$PWD/ClientGuiConfig.ini)

Linux try_cmpxchg mysterious inline assembly

I need some help understanding Linux's `try_cmpxchg semantics and implementation. In the kernel source, it is implemented as:
#define __raw_try_cmpxchg(_ptr, _pold, _new, size, lock) \
({ \
bool success; \
__typeof__(_ptr) _old = (_pold); \
__typeof__(*(_ptr)) __old = *_old; \
__typeof__(*(_ptr)) __new = (_new); \
switch (size) { \
case __X86_CASE_B: \
{ \
volatile u8 *__ptr = (volatile u8 *)(_ptr); \
asm volatile(lock "cmpxchgb %[new], %[ptr]" \
CC_SET(z) \
: CC_OUT(z) (success), \
[ptr] "+m" (*__ptr), \
[old] "+a" (__old) \
: [new] "q" (__new) \
: "memory"); \
break; \
} \
case __X86_CASE_W: \
{ \
volatile u16 *__ptr = (volatile u16 *)(_ptr); \
asm volatile(lock "cmpxchgw %[new], %[ptr]" \
CC_SET(z) \
: CC_OUT(z) (success), \
[ptr] "+m" (*__ptr), \
[old] "+a" (__old) \
: [new] "r" (__new) \
: "memory"); \
break; \
} \
case __X86_CASE_L: \
{ \
volatile u32 *__ptr = (volatile u32 *)(_ptr); \
asm volatile(lock "cmpxchgl %[new], %[ptr]" \
CC_SET(z) \
: CC_OUT(z) (success), \
[ptr] "+m" (*__ptr), \
[old] "+a" (__old) \
: [new] "r" (__new) \
: "memory"); \
break; \
} \
case __X86_CASE_Q: \
{ \
volatile u64 *__ptr = (volatile u64 *)(_ptr); \
asm volatile(lock "cmpxchgq %[new], %[ptr]" \
CC_SET(z) \
: CC_OUT(z) (success), \
[ptr] "+m" (*__ptr), \
[old] "+a" (__old) \
: [new] "r" (__new) \
: "memory"); \
break; \
} \
default: \
__cmpxchg_wrong_size(); \
} \
if (unlikely(!success)) \
*_old = __old; \
likely(success); \
})
#define __try_cmpxchg(ptr, pold, new, size) \
__raw_try_cmpxchg((ptr), (pold), (new), (size), LOCK_PREFIX)
#define try_cmpxchg(ptr, pold, new) \
__try_cmpxchg((ptr), (pold), (new), sizeof(*(ptr)))
I am curious what those CC_SET and CC_OUT means. They are defined as:
/*
* Macros to generate condition code outputs from inline assembly,
* The output operand must be type "bool".
*/
#ifdef __GCC_ASM_FLAG_OUTPUTS__
# define CC_SET(c) "\n\t/* output condition code " #c "*/\n"
# define CC_OUT(c) "=#cc" #c
#else
# define CC_SET(c) "\n\tset" #c " %[_cc_" #c "]\n"
# define CC_OUT(c) [_cc_ ## c] "=qm"
#endif
Also, it would be great if you can explain the exact semantics of try_cmpxchg (not quite understand how can a atomic cmpxchg fail...)
Newer versions of gcc (I believe from version 6) support specific flag outputs. The macros are there to use this support if available, else fall back to the old way by doing a setCC instruction and a temporary output.
As to how cmpxchg can "fail": it does a compare so it fails if that compare fails, in which case the destination is unchanged and the current value is fetched from memory. Consult an instruction set reference for the details.

Some kernel ARM code

I was reading through some ARM kernel sources till I stumbled upon the following function :-
314 #define __get_user_asm_byte(x, addr, err) \
315 __asm__ __volatile__( \
316 "1: " TUSER(ldrb) " %1,[%2],#0\n" \
317 "2:\n" \
318 " .pushsection .fixup,\"ax\"\n" \
319 " .align 2\n" \
320 "3: mov %0, %3\n" \
321 " mov %1, #0\n" \
322 " b 2b\n" \
323 " .popsection\n" \
324 " .pushsection __ex_table,\"a\"\n" \
325 " .align 3\n" \
326 " .long 1b, 3b\n" \
327 " .popsection" \
328 : "+r" (err), "=&r" (x) \
329 : "r" (addr), "i" (-EFAULT) \
330 : "cc")
The calling context seems is as follows :-
299 #define __get_user_err(x, ptr, err) \
300 do { \
301 unsigned long __gu_addr = (unsigned long)(ptr); \
302 unsigned long __gu_val; \
303 __chk_user_ptr(ptr); \
304 might_fault(); \
305 switch (sizeof(*(ptr))) { \
306 case 1: __get_user_asm_byte(__gu_val, __gu_addr, err); break; \
Now I have a few doubts I'd like to clear about the ARM assembly above.
What is the function __get_user_asm_byte doing? I can see that r3 is copied into r0 and that the value 0 is moved into r1. After that does it branches to the offset 0x2b?
What is the function trying to do? What does the "+r" (err), "=&r" (x) from line 328 onward mean?
What is a .pushsection and a .popsection?
Why does ARM assembly thats written for the kernel so different syntactically(what's the assembler used? Why are there %<regno> instead of r<regno>?)
First, as comments have noted, the syntax is standard gcc inline assembly syntax (the +r, =&r, %<arg> parts).
The rest is kernel magic designed to handle page faults. The point of get_user_asm_byte is to pull a byte from user-space. However, in pulling data from user-space, two special situations need to be accommodated:
A perfectly legitimate user-space address that is simply not present at the moment (i.e. usually because it's paged out)
An illegal user-space address.
Either could cause a page fault. For (1), the desired behavior is to restore the user page (read it back in from swap space or whatever), then re-execute the load instruction, resulting in eventual success; for (2), the desired behavior is to fail the operation without retrying, returning an EFAULT error to the caller.
Without special handling, the normal behavior of a page fault in kernel mode is an "Oops". The special sections coordinate with the page fault handling code and make it possible to recover correctly. If you want to understand the details of how this works, search for __ex_table in the kernel source.
Also, checkout kernel doc at: Documentation/x86/exception-tables.txt
The only ARM specific item is to use either ldrt or MMU domains; this is conditional in domain.h. Modern ARM CPUs support domains. The translated load/store variants apply a user mode access for older CPUs.

Sphinx search gives no results

I have a new index on a new table which is not returning any results. This is very odd, and I've never run into this problem before. Other indices (which are built almost identically) are searchable just fine from the search CLI and my API.
Here's my conf file
source topicalindex
{
type = pgsql
sql_host = localhost
sql_user = user
sql_pass = password
sql_db = db
sql_port = 5432 # optional, default is 3306
sql_query = SELECT id, topic, reference, start_ref, end_ref, see_also_html FROM my_topicalindex
sql_attr_uint = topic
sql_attr_uint = reference
}
index topicalindex_index
{
source = topicalindex
path = /path/to/data/topical_index
docinfo = extern
charset_type = utf-8
}
indexer
{
mem_limit = 32M
}
searchd
{
listen = 3312
log = /path/to/searchd.log
query_log = /path/to/query.log
read_timeout = 5
max_children = 30
pid_file = /usr/local/var/searchd.pid
max_matches = 30000
seamless_rotate = 1
preopen_indexes = 0
unlink_old = 1
}
Here's a excerpt proving there's content in the DB
[myself]:myapp(master)$ psql -d mydb -h localhost
psql (9.2.2)
Type "help" for help.
esv=# SELECT * FROM my_topicalindex LIMIT 1;
id | topic | reference | start_ref | end_ref | see_also_html
------+---------+--------------------+-----------+---------+---------------
2810 | Abraham | Genesis chs. 11–25 | 1011001 | 1025034 | blank
(1 row)
Here's the indexing process:
$ indexer --rotate --all --config /path/to/sphinx-topical.conf
Sphinx 0.9.9-rc2 (r1785)
Copyright (c) 2001-2009, Andrew Aksyonoff
using config file '/path/to/sphinx-topical.conf'...
indexing index 'topicalindex_index'...
collected 2809 docs, 0.1 MB
sorted 0.0 Mhits, 100.0% done
total 2809 docs, 75007 bytes
total 0.067 sec, 1117456 bytes/sec, 41848.54 docs/sec
total 3 reads, 0.000 sec, 47.0 kb/call avg, 0.0 msec/call avg
total 7 writes, 0.000 sec, 37.0 kb/call avg, 0.0 msec/call avg
rotating indices: succesfully sent SIGHUP to searchd (pid=79833).
And the files showing they have content
[myself]:myapp(master)$ ll /path/to/data/
total 160
drwxr-xr-x 10 myself admin 340 Aug 29 08:56 ./
drwxr-xr-x 3 myself admin 102 Jun 1 2012 ../
-rw-r--r-- 1 myself admin 33708 Aug 29 08:56 topical_index.spa
-rw-r--r-- 1 myself admin 51538 Aug 29 08:56 topical_index.spd
-rw-r--r-- 1 myself admin 326 Aug 29 08:56 topical_index.sph
-rw-r--r-- 1 myself admin 15721 Aug 29 08:56 topical_index.spi
-rw-r--r-- 1 myself admin 0 Aug 29 08:56 topical_index.spk
-rw------- 1 myself admin 0 Aug 29 08:56 topical_index.spl
-rw-r--r-- 1 myself admin 0 Aug 29 08:56 topical_index.spm
-rw-r--r-- 1 myself admin 52490 Aug 29 08:56 topical_index.spp
and then -- my search with 0 results
[myself]:myapp(master)$ search -i topicalindex_index -a "Abraham"
Sphinx 0.9.9-rc2 (r1785)
Copyright (c) 2001-2009, Andrew Aksyonoff
using config file '/usr/local/etc/sphinx.conf'...
index 'topicalindex_index': query 'Abraham ': returned 0 matches of 0 total in 0.000 sec
words:
1. 'abraham': 0 documents, 0 hits
Why am I getting 0 results when I search "Abraham". (Strangely, I do get results when searching "a")
Is this a problem with my conf file? Is it something else?
EDIT
I've noticed that when searching "a" it is only searching against those rows which do have content for the see_also_html field. The number of matches matches the count of fields which have data for each column.
sql_query = SELECT id, topic, reference, start_ref, end_ref, see_also_html FROM my_topicalindex
sql_attr_uint = topic
sql_attr_uint = reference
id | topic | reference | start_ref | end_ref | see_also_html
------+---------+--------------------+-----------+---------+---------------
2810 | Abraham | Genesis chs. 11–25 | 1011001 | 1025034 | blank
You've made topic and reference integer attributes. (So they won't be included as full-text fields)
Columns not made an attribute (excluding the first column!) are automatically fields.
So start_ref, end_ref and see_also_html will be full-text fields, and therefore searchable.
OUT OF TOPIC ANSWER
Note: for more readability in your file
you can replace
sql_query = SELECT id, topic, reference, start_ref, end_ref, see_also_html FROM my_topicalindex
by
sql_query = SELECT id, \
topic, \
reference, \
start_ref, \
end_ref, \
see_also_html \
FROM my_topicalindex
Just be careful \ would be the last character (not space)
Sample of statement where I'm happy to use this notation
sql_query = \
SELECT /* sphinx index article Video */ \
`article`.id AS id, \
UNIX_TIMESTAMP(`article`.fromDate) AS ressource_date, \
unix_timestamp(now()) AS indexing_date, \
`programs`.id AS emission_id, \
IFnull(cat.parent_id, cat.id) AS genre_id, \
`article`.`title` AS `title_str`, \
`article`.`title` AS `title`, \
`article`.`feed` AS `feed_ordinal`, \
`article`.`feed` AS `feed`, \
`article`.`category` AS `category`, \
`article`.`dossierId` AS `dossierId`, \
`article`.`mainImage` AS `mainImage`, \
group_concat(`keywords`.`translation`) AS `tags`, \
group_concat(o.id) AS `video_id_list`, \
concat(' summary:', \
`article`.summary, \
' synopsis', \
`article`.synopsis, \
' Summary Title :', \
`article`.summaryTitle, \
' Url', \
`article`.title_url, \
' Main', \
`article`.mainParagraph) AS content, \
concat( ' Signature:', \
`article`.signature) AS subcontent, \
UNIX_TIMESTAMP(article.fromDate) AS online_from_date, \
if ( UNIX_TIMESTAMP(article.toDate)<2000000000, \
UNIX_TIMESTAMP(article.toDate), \
2000000000) AS online_to_date \
FROM \
`article`.`article` \
LEFT JOIN \
`media`.`programs` \
ON \
`article`.`category` = `programs`.`articleReferenceCategory` \
LEFT JOIN \
`media`.`category` AS cat \
ON `programs`.mainCategoryId = cat.id \
LEFT JOIN \
article.article_keyword AS ak \
ON \
ak.articleId = article.id \
LEFT JOIN \
article.keyword AS keywords \
ON \
ak.keywordId = keywords.id \
AND `keywords`.`visible` = 1 \
AND `keywords`.`translation` !='' \
LEFT JOIN \
article.embed AS ae \
ON \
ae.articleId = article.id \
INNER JOIN \
media.`objects` AS o \
ON \
o.id = SUBSTRING_INDEX( SUBSTRING(ae.code, 5+LOCATE('rel="',ae.code)), '"', 1) \
AND o.type='video' \
AND o.status='complete' \
AND o.active=1 \
AND ( o.part_type IS NULL OR o.part_type = 'extract') \
AND o.wildcard not like 'www.%.be/presse%' \
\
WHERE \
`article`.`validated` = 1 \
AND `article`.`published` = 1 \
AND `article`.`deleted` = 0 \
AND `article`.`displayDate` < Now() \
AND `article`.`fromDate` < Now() \
AND `article`.`toDate` > Now() \
AND `article`.id >= $start \
AND `article`.id < $end \
GROUP BY `article`.id

Resources