Ubuntu 14.04 preseed LVM disk config - linux

I'm having some issues getting my partitions to be of type primary, and not logical/extended.
Here is the relevant code in my preseed:
d-i partman-auto/disk string /dev/sda
d-i partman-auto/method string lvm
d-i partman-lvm/device_remove_lvm boolean true
d-i partman-md/device_remove_md boolean true
d-i partman-lvm/confirm boolean true
d-i partman-auto-lvm/guided_size string max
d-i partman-auto/choose_recipe select boot-root
d-i partman-auto-lvm/new_vg_name string vg00
d-i partman-auto/expert_recipe string \
boot-root :: \
512 512 512 ext3 \
$primary{ } $bootable{ } \
method{ format } format{ } \
use_filesystem{ } filesystem{ ext3 } \
mountpoint{ /boot } \
. \
2048 2048 2048 swap \
$primary{ } $lvmok{ } lv_name{ lv_swap } $defaultignore{} \
method{ swap } format{ } \
. \
1024 10000 -1 ext4 \
$primary{ } $lvmok{ } lv_name{ lv_root } $defaultignore{}\
method{ format } format{ } \
use_filesystem{ } filesystem{ ext4 } \
mountpoint{ / } \
.
d-i partman-partitioning/confirm_write_new_label boolean true
d-i partman/choose_partition select finish
d-i partman/confirm boolean true
d-i partman/confirm_nooverwrite boolean true
d-i partman-lvm/confirm_nooverwrite boolean true
The problem is, this then creates the following partition scheme:
root#ubuntu-server-1404-devit:~# fdisk /dev/sda
Command (m for help): p
Disk /dev/sda: 10.7 GB, 10737418240 bytes
255 heads, 63 sectors/track, 1305 cylinders, total 20971520 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x0009ac4d
Device Boot Start End Blocks Id System
/dev/sda1 * 2048 2000895 999424 83 Linux
/dev/sda2 2002942 20969471 9483265 5 Extended
/dev/sda5 2002944 20969471 9483264 8e Linux LVM
I'd like to remove this unnecessary Extended / logical partition, and just have the Linux LVM partition be on sda2 (primary). Like so:
Device Boot Start End Blocks Id System
/dev/sda1 * 2048 2000895 999424 83 Linux
/dev/sda2 2002942 20969471 9483265 8e Linux LVM

Well the documentation for this is pretty embarassingly minimal but I've figured it out (needed to define the volume group name and method type lvm for some reason):
d-i partman-auto/disk string /dev/sda
d-i partman-auto/method string lvm
d-i partman-lvm/device_remove_lvm boolean true
d-i partman-md/device_remove_md boolean true
d-i partman-lvm/confirm boolean true
d-i partman-auto-lvm/guided_size string max
d-i partman-auto/choose_recipe select boot-root
d-i partman-auto-lvm/new_vg_name string vg00
d-i partman-auto/expert_recipe string \
boot-root :: \
1024 1024 1024 ext4 \
$primary{ } $bootable{ } \
method{ format } format{ } \
use_filesystem{ } filesystem{ ext4 } \
mountpoint{ /boot } \
. \
100 1000 1000000000 ext4 \
$defaultignore{ } \
$primary{ } \
method{ lvm } \
device{ /dev/sda } \
vg_name{ vg00 } \
. \
2048 2048 2048 swap \
$lvmok{ } lv_name{ lv_swap } in_vg{ vg00 } \
method{ swap } format{ } \
. \
1024 3072 -1 ext4 \
$lvmok{} lv_name{ lv_root } in_vg{ vg00 } \
method{ format } format{ } \
use_filesystem{ } filesystem{ ext4 } \
mountpoint{ / } \
.
d-i partman-partitioning/confirm_write_new_label boolean true
d-i partman/choose_partition select finish
d-i partman/confirm boolean true
d-i partman/confirm_nooverwrite boolean true
d-i partman-lvm/confirm_nooverwrite boolean true

Related

How to craete a NVMe with 4K sector size in KVM-QEMU

For testing purposes, I am creating a virtual machine as it follows:
qemu-img create -f qcow2 \
-o cluster_size=4096,preallocation=full \
/home/marcop/.libvirt/nvme-20G.qcow2 20G
qemu-system-x86_64 -machine q35,accel=kvm \
-m 4096 \
-smp 4 \
-cpu host \
-boot d \
-cdrom /var/lib/libvirt/isos/archlinux-2020.10.01-x86_64.iso \
-drive file=/home/marcop/.libvirt/nvme-20G.qcow2,if=none,aio=native,cache.direct=on,id=D24 \
-device nvme,drive=D24,serial=1234,logical_block_size=4096,physical_block_size=4096
When booted inside the machine, I use fdisk and nvme-cli to check the sector size, but it's always 512B.
pacman -Sy nvme-cli
fdisk -l /dev/nvme0n1
with output:
Disk /dev/nvme0n1: 20 GiB, 21474836480 bytes, 41943040 sectors
Disk model: QEMU NVMe Ctrl
Units: sectors of 1 * 512 = 512 bytes
Sector size (logica/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Looking for the available sector size modes with nvme-cli (see here and here for details on NVMe)
nvme id-ns /dev/nvme0n1
return output:
NVME Identify Namespace 1:
nsze : 0x2800000
ncap : 0x2800000
nuse : 0x2800000
nsfeat : 0
nlbaf : 0
flbas : 0
mc : 0
dpc : 0
dps : 0
nmic : 0
rescap : 0
fpi : 0
dlfeat : 0
nawun : 0
nawupf : 0
nacwu : 0
nabsn : 0
nabo : 0
nabspf : 0
noiob : 0
nvmcap : 0
nsattr : 0
nvmsetid: 0
anagrpid: 0
endgid : 0
nguid : 00000000000000000000000000000000
eui64 : 0000000000000000
lbaf 0 : ms:0 lbads:9 rp:0 (in use)
Which indicates the only one sector profile exits. For comparison, the output of the same command issued for my physical NVMe returns
[...]
lbaf 0 : ms:0 lbads:9 rp:0x2 (in use)
lbaf 1 : ms:0 lbads:12 rp:0x1
Any help would be greatly appreciated!
run command to format with 4k. But beware that format will wipe out data.
nvme format /dev/nvme0n1 --lbaf=1

Linux try_cmpxchg mysterious inline assembly

I need some help understanding Linux's `try_cmpxchg semantics and implementation. In the kernel source, it is implemented as:
#define __raw_try_cmpxchg(_ptr, _pold, _new, size, lock) \
({ \
bool success; \
__typeof__(_ptr) _old = (_pold); \
__typeof__(*(_ptr)) __old = *_old; \
__typeof__(*(_ptr)) __new = (_new); \
switch (size) { \
case __X86_CASE_B: \
{ \
volatile u8 *__ptr = (volatile u8 *)(_ptr); \
asm volatile(lock "cmpxchgb %[new], %[ptr]" \
CC_SET(z) \
: CC_OUT(z) (success), \
[ptr] "+m" (*__ptr), \
[old] "+a" (__old) \
: [new] "q" (__new) \
: "memory"); \
break; \
} \
case __X86_CASE_W: \
{ \
volatile u16 *__ptr = (volatile u16 *)(_ptr); \
asm volatile(lock "cmpxchgw %[new], %[ptr]" \
CC_SET(z) \
: CC_OUT(z) (success), \
[ptr] "+m" (*__ptr), \
[old] "+a" (__old) \
: [new] "r" (__new) \
: "memory"); \
break; \
} \
case __X86_CASE_L: \
{ \
volatile u32 *__ptr = (volatile u32 *)(_ptr); \
asm volatile(lock "cmpxchgl %[new], %[ptr]" \
CC_SET(z) \
: CC_OUT(z) (success), \
[ptr] "+m" (*__ptr), \
[old] "+a" (__old) \
: [new] "r" (__new) \
: "memory"); \
break; \
} \
case __X86_CASE_Q: \
{ \
volatile u64 *__ptr = (volatile u64 *)(_ptr); \
asm volatile(lock "cmpxchgq %[new], %[ptr]" \
CC_SET(z) \
: CC_OUT(z) (success), \
[ptr] "+m" (*__ptr), \
[old] "+a" (__old) \
: [new] "r" (__new) \
: "memory"); \
break; \
} \
default: \
__cmpxchg_wrong_size(); \
} \
if (unlikely(!success)) \
*_old = __old; \
likely(success); \
})
#define __try_cmpxchg(ptr, pold, new, size) \
__raw_try_cmpxchg((ptr), (pold), (new), (size), LOCK_PREFIX)
#define try_cmpxchg(ptr, pold, new) \
__try_cmpxchg((ptr), (pold), (new), sizeof(*(ptr)))
I am curious what those CC_SET and CC_OUT means. They are defined as:
/*
* Macros to generate condition code outputs from inline assembly,
* The output operand must be type "bool".
*/
#ifdef __GCC_ASM_FLAG_OUTPUTS__
# define CC_SET(c) "\n\t/* output condition code " #c "*/\n"
# define CC_OUT(c) "=#cc" #c
#else
# define CC_SET(c) "\n\tset" #c " %[_cc_" #c "]\n"
# define CC_OUT(c) [_cc_ ## c] "=qm"
#endif
Also, it would be great if you can explain the exact semantics of try_cmpxchg (not quite understand how can a atomic cmpxchg fail...)
Newer versions of gcc (I believe from version 6) support specific flag outputs. The macros are there to use this support if available, else fall back to the old way by doing a setCC instruction and a temporary output.
As to how cmpxchg can "fail": it does a compare so it fails if that compare fails, in which case the destination is unchanged and the current value is fetched from memory. Consult an instruction set reference for the details.

linux lvm partition using partman in preseed

I'm trying to customise preseed on ubuntu 14.04. Where all the parameter required for installation are stored. At the time of first OS boot value and variables are exported, and configuration is completed without any manual intervention .
During this process for hardware configuration i opted for LVM based architecture.
When i tried with fixed disk space it works fine for below configuration
d-i partman-auto/expert_recipe string \
boot-root :: \
250 1000 300 ext4 \
$primary{ } $bootable{ } \
method{ format } format{ } \
use_filesystem{ } filesystem{ ext4 } \
mountpoint{ /boot } \
. \
# swap with at least 5GB, maximum size of memory
5120 500 100% linux-swap \
$lvmok{ } \
method{ swap } format{ } \
. \
# /var/db/application with at least 40GB, maximum 40GB, and ext4
40960 1000 40960 ext4 \
$lvmok{ } \
method{ format } format{ } \
use_filesystem{ } filesystem{ ext4 } \
mountpoint{ /var/db/application } \
. \
# / with at least 40GB, maximum 40GB, and ext4
40960 1000 40960 ext4 \
$lvmok{ } \
method{ format } format{ } \
use_filesystem{ } filesystem{ ext4 } \
mountpoint{ / } \
. \
# /var/log/application with at least 20GB, maximum rest of space, and ext4
20480 1000 1000000 ext4 \
$lvmok{ } \
method{ format } format{ } \
use_filesystem{ } filesystem{ ext4 } \
mountpoint{ /var/log/application } \
. \
My query is when i tried assigning disk space based on percentage, Since hard disk allocation is different every time. It doesn't work for me. The partman configuration is as below
d-i partman-auto/expert_recipe string \
boot-root :: \
250 1000 300 ext4 \
$primary{ } $bootable{ } \
method{ format } format{ } \
use_filesystem{ } filesystem{ ext4 } \
mountpoint{ /boot } \
. \
# swap with at least 5GB, maximum size of memory
5120 500 100% linux-swap \
$lvmok{ } \
method{ swap } format{ } \
. \
# /var/db/application with at least 40% of disk provided, maximum 40% of disk provided, and ext4
40% 1000 40% ext4 \
$lvmok{ } \
method{ format } format{ } \
use_filesystem{ } filesystem{ ext4 } \
mountpoint{ /var/db/application } \
. \
# / with at least 35% of disk provided, maximum 35% of disk provided, and ext4
35% 1000 35% ext4 \
$lvmok{ } \
method{ format } format{ } \
use_filesystem{ } filesystem{ ext4 } \
mountpoint{ / } \
. \
# /var/log/application with at least 20% of disk provided, maximum rest of space, and ext4
20% 1000 1000000 ext4 \
$lvmok{ } \
method{ format } format{ } \
use_filesystem{ } filesystem{ ext4 } \
mountpoint{ /var/log/application } \
. \
Any help where i'm going wrong

How is the init process started in the Linux kernel?

I am trying to understand the init process in the linux kernel which is the first process and is statically initialized with the INIT_TASK macro.
161 #define INIT_TASK(tsk) \
162 { \
163 .state = 0, \
164 .stack = &init_thread_info, \
165 .usage = ATOMIC_INIT(2), \
166 .flags = PF_KTHREAD, \
167 .prio = MAX_PRIO-20, \
168 .static_prio = MAX_PRIO-20, \
169 .normal_prio = MAX_PRIO-20, \
170 .policy = SCHED_NORMAL, \
171 .cpus_allowed = CPU_MASK_ALL, \
172 .nr_cpus_allowed= NR_CPUS, \
173 .mm = NULL, \
174 .active_mm = &init_mm, \
175 .se = { \
176 .group_node = LIST_HEAD_INIT(tsk.se.group_node), \
177 }, \
178 .rt = { \
179 .run_list = LIST_HEAD_INIT(tsk.rt.run_list), \
180 .time_slice = RR_TIMESLICE, \
181 }, \
182 .tasks = LIST_HEAD_INIT(tsk.tasks), \
183 INIT_PUSHABLE_TASKS(tsk) \
184 INIT_CGROUP_SCHED(tsk) \
185 .ptraced = LIST_HEAD_INIT(tsk.ptraced), \
186 .ptrace_entry = LIST_HEAD_INIT(tsk.ptrace_entry), \
187 .real_parent = &tsk, \
188 .parent = &tsk, \
189 .children = LIST_HEAD_INIT(tsk.children), \
190 .sibling = LIST_HEAD_INIT(tsk.sibling), \
191 .group_leader = &tsk, \
192 RCU_POINTER_INITIALIZER(real_cred, &init_cred), \
193 RCU_POINTER_INITIALIZER(cred, &init_cred), \
194 .comm = INIT_TASK_COMM, \
195 .thread = INIT_THREAD, \
196 .fs = &init_fs, \
197 .files = &init_files, \
198 .signal = &init_signals, \
199 .sighand = &init_sighand, \
200 .nsproxy = &init_nsproxy, \
201 .pending = { \
202 .list = LIST_HEAD_INIT(tsk.pending.list), \
203 .signal = {{0}}}, \
204 .blocked = {{0}}, \
205 .alloc_lock = __SPIN_LOCK_UNLOCKED(tsk.alloc_lock), \
206 .journal_info = NULL, \
207 .cpu_timers = INIT_CPU_TIMERS(tsk.cpu_timers), \
208 .pi_lock = __RAW_SPIN_LOCK_UNLOCKED(tsk.pi_lock), \
209 .timer_slack_ns = 50000, /* 50 usec default slack */ \
210 .pids = { \
211 [PIDTYPE_PID] = INIT_PID_LINK(PIDTYPE_PID), \
212 [PIDTYPE_PGID] = INIT_PID_LINK(PIDTYPE_PGID), \
213 [PIDTYPE_SID] = INIT_PID_LINK(PIDTYPE_SID), \
214 }, \
215 .thread_group = LIST_HEAD_INIT(tsk.thread_group), \
216 INIT_IDS \
217 INIT_PERF_EVENTS(tsk) \
218 INIT_TRACE_IRQFLAGS \
219 INIT_LOCKDEP \
220 INIT_FTRACE_GRAPH \
221 INIT_TRACE_RECURSION \
222 INIT_TASK_RCU_PREEMPT(tsk) \
223 INIT_CPUSET_SEQ \
224 INIT_VTIME(tsk) \
225 }
But I am not able to figure out
how it will be executed?
Where it is scheduled and
which lines of code in the linux kernel start executing immediately when we say we have scheduled this init_task task? Is there any function which it calls?
The kernel calls "init" as one of the very last things it does during kernel initialization. The function kernel_init() in init/main.c has the logic.
You will notice that the kernel tries four different combinations of init, and expects one of them to succeed. You will also notice that you can override what the kernel executes on startup by feeding the kernel command line parameter "init". So, you can say, for example, init=/bin/mystartup on the kernel command line and start your own custom application instead of the default /sbin/init. Notice also that on most modern systems, even embedded systems, /sbin/init is a soft link that points to the real executable.
To more generally answer your question, study this source file (main.c) you can see virtually all of the details of Linux kernel initialization, after the low-level assembly stuff and platform initialization, which, beyond the educational value, you shouldn't have to touch nor care about much.
The main mechanism is to call do_execve() with fixed arguments of argv_init and envp_init. The elf file is parsed and initial program counter (PC) is set as per the file. All memory management (mm) pages are mapped to the disks backing store. The code is set to run. On the initial PC fetch when it is scheduled, a page fault is generated which reads the first code page into memory. This is the same as any other execve() call.

Sphinx search gives no results

I have a new index on a new table which is not returning any results. This is very odd, and I've never run into this problem before. Other indices (which are built almost identically) are searchable just fine from the search CLI and my API.
Here's my conf file
source topicalindex
{
type = pgsql
sql_host = localhost
sql_user = user
sql_pass = password
sql_db = db
sql_port = 5432 # optional, default is 3306
sql_query = SELECT id, topic, reference, start_ref, end_ref, see_also_html FROM my_topicalindex
sql_attr_uint = topic
sql_attr_uint = reference
}
index topicalindex_index
{
source = topicalindex
path = /path/to/data/topical_index
docinfo = extern
charset_type = utf-8
}
indexer
{
mem_limit = 32M
}
searchd
{
listen = 3312
log = /path/to/searchd.log
query_log = /path/to/query.log
read_timeout = 5
max_children = 30
pid_file = /usr/local/var/searchd.pid
max_matches = 30000
seamless_rotate = 1
preopen_indexes = 0
unlink_old = 1
}
Here's a excerpt proving there's content in the DB
[myself]:myapp(master)$ psql -d mydb -h localhost
psql (9.2.2)
Type "help" for help.
esv=# SELECT * FROM my_topicalindex LIMIT 1;
id | topic | reference | start_ref | end_ref | see_also_html
------+---------+--------------------+-----------+---------+---------------
2810 | Abraham | Genesis chs. 11–25 | 1011001 | 1025034 | blank
(1 row)
Here's the indexing process:
$ indexer --rotate --all --config /path/to/sphinx-topical.conf
Sphinx 0.9.9-rc2 (r1785)
Copyright (c) 2001-2009, Andrew Aksyonoff
using config file '/path/to/sphinx-topical.conf'...
indexing index 'topicalindex_index'...
collected 2809 docs, 0.1 MB
sorted 0.0 Mhits, 100.0% done
total 2809 docs, 75007 bytes
total 0.067 sec, 1117456 bytes/sec, 41848.54 docs/sec
total 3 reads, 0.000 sec, 47.0 kb/call avg, 0.0 msec/call avg
total 7 writes, 0.000 sec, 37.0 kb/call avg, 0.0 msec/call avg
rotating indices: succesfully sent SIGHUP to searchd (pid=79833).
And the files showing they have content
[myself]:myapp(master)$ ll /path/to/data/
total 160
drwxr-xr-x 10 myself admin 340 Aug 29 08:56 ./
drwxr-xr-x 3 myself admin 102 Jun 1 2012 ../
-rw-r--r-- 1 myself admin 33708 Aug 29 08:56 topical_index.spa
-rw-r--r-- 1 myself admin 51538 Aug 29 08:56 topical_index.spd
-rw-r--r-- 1 myself admin 326 Aug 29 08:56 topical_index.sph
-rw-r--r-- 1 myself admin 15721 Aug 29 08:56 topical_index.spi
-rw-r--r-- 1 myself admin 0 Aug 29 08:56 topical_index.spk
-rw------- 1 myself admin 0 Aug 29 08:56 topical_index.spl
-rw-r--r-- 1 myself admin 0 Aug 29 08:56 topical_index.spm
-rw-r--r-- 1 myself admin 52490 Aug 29 08:56 topical_index.spp
and then -- my search with 0 results
[myself]:myapp(master)$ search -i topicalindex_index -a "Abraham"
Sphinx 0.9.9-rc2 (r1785)
Copyright (c) 2001-2009, Andrew Aksyonoff
using config file '/usr/local/etc/sphinx.conf'...
index 'topicalindex_index': query 'Abraham ': returned 0 matches of 0 total in 0.000 sec
words:
1. 'abraham': 0 documents, 0 hits
Why am I getting 0 results when I search "Abraham". (Strangely, I do get results when searching "a")
Is this a problem with my conf file? Is it something else?
EDIT
I've noticed that when searching "a" it is only searching against those rows which do have content for the see_also_html field. The number of matches matches the count of fields which have data for each column.
sql_query = SELECT id, topic, reference, start_ref, end_ref, see_also_html FROM my_topicalindex
sql_attr_uint = topic
sql_attr_uint = reference
id | topic | reference | start_ref | end_ref | see_also_html
------+---------+--------------------+-----------+---------+---------------
2810 | Abraham | Genesis chs. 11–25 | 1011001 | 1025034 | blank
You've made topic and reference integer attributes. (So they won't be included as full-text fields)
Columns not made an attribute (excluding the first column!) are automatically fields.
So start_ref, end_ref and see_also_html will be full-text fields, and therefore searchable.
OUT OF TOPIC ANSWER
Note: for more readability in your file
you can replace
sql_query = SELECT id, topic, reference, start_ref, end_ref, see_also_html FROM my_topicalindex
by
sql_query = SELECT id, \
topic, \
reference, \
start_ref, \
end_ref, \
see_also_html \
FROM my_topicalindex
Just be careful \ would be the last character (not space)
Sample of statement where I'm happy to use this notation
sql_query = \
SELECT /* sphinx index article Video */ \
`article`.id AS id, \
UNIX_TIMESTAMP(`article`.fromDate) AS ressource_date, \
unix_timestamp(now()) AS indexing_date, \
`programs`.id AS emission_id, \
IFnull(cat.parent_id, cat.id) AS genre_id, \
`article`.`title` AS `title_str`, \
`article`.`title` AS `title`, \
`article`.`feed` AS `feed_ordinal`, \
`article`.`feed` AS `feed`, \
`article`.`category` AS `category`, \
`article`.`dossierId` AS `dossierId`, \
`article`.`mainImage` AS `mainImage`, \
group_concat(`keywords`.`translation`) AS `tags`, \
group_concat(o.id) AS `video_id_list`, \
concat(' summary:', \
`article`.summary, \
' synopsis', \
`article`.synopsis, \
' Summary Title :', \
`article`.summaryTitle, \
' Url', \
`article`.title_url, \
' Main', \
`article`.mainParagraph) AS content, \
concat( ' Signature:', \
`article`.signature) AS subcontent, \
UNIX_TIMESTAMP(article.fromDate) AS online_from_date, \
if ( UNIX_TIMESTAMP(article.toDate)<2000000000, \
UNIX_TIMESTAMP(article.toDate), \
2000000000) AS online_to_date \
FROM \
`article`.`article` \
LEFT JOIN \
`media`.`programs` \
ON \
`article`.`category` = `programs`.`articleReferenceCategory` \
LEFT JOIN \
`media`.`category` AS cat \
ON `programs`.mainCategoryId = cat.id \
LEFT JOIN \
article.article_keyword AS ak \
ON \
ak.articleId = article.id \
LEFT JOIN \
article.keyword AS keywords \
ON \
ak.keywordId = keywords.id \
AND `keywords`.`visible` = 1 \
AND `keywords`.`translation` !='' \
LEFT JOIN \
article.embed AS ae \
ON \
ae.articleId = article.id \
INNER JOIN \
media.`objects` AS o \
ON \
o.id = SUBSTRING_INDEX( SUBSTRING(ae.code, 5+LOCATE('rel="',ae.code)), '"', 1) \
AND o.type='video' \
AND o.status='complete' \
AND o.active=1 \
AND ( o.part_type IS NULL OR o.part_type = 'extract') \
AND o.wildcard not like 'www.%.be/presse%' \
\
WHERE \
`article`.`validated` = 1 \
AND `article`.`published` = 1 \
AND `article`.`deleted` = 0 \
AND `article`.`displayDate` < Now() \
AND `article`.`fromDate` < Now() \
AND `article`.`toDate` > Now() \
AND `article`.id >= $start \
AND `article`.id < $end \
GROUP BY `article`.id

Resources