Gdb Crash Dump Guide

Analysing Crash Dumps

Initiating GDB

Command:

Prompt $ sudo gdb <executable> <corefile>

E.g.

prompt$ sudo gdb ./ipperte core

GNU gdb (Ubuntu 7.7-0ubuntu3.1) 7.7
Copyright (C) 2014 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from ./ipperte...done.
[New LWP 10122]
[New LWP 10062]
[New LWP 10063]
[New LWP 10064]
[New LWP 10065]
[New LWP 10066]
[New LWP 10067]
[New LWP 10068]
[New LWP 10125]
[New LWP 10059]
[New LWP 10061]
[New LWP 10071]
[New LWP 10072]
[New LWP 10120]
[New LWP 10070]

warning: Could not load shared library symbols for /tmp/xf-33417f81c8f3d65ff8f8a781917b9932.tmp.
Do you need "set solib-search-path" or "set sysroot"?
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Core was generated by `./ipperte'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0 0x00007fd5ffb4e8f3 in vfprintf () from /lib/x86_64-linux-gnu/libc.so.6

Here we can see we died with a SIGSEGV, Segmentation fault in /lib/x86_64-linux-gnu/libc.so.6

Visual Mode

Or use the additional switch -tui to enter visual mode e.g.

sudo gdb -tui ./ipperte core

This produces a split screen text mode interface where the top part is source code and the lower half is a command window accepting standard gdb commands

Using select <n> will switch to stack frame N and handily show the source associated with that frame.

If it’s a Segfault - what’s the referenced address:

The _siginfo structure provides this

(gdb) print $_siginfo._sifields._sigfault.si_addr

Looking at the stack (backtrace)

Command:

(gdb) bt

Or

(gdb) backtrace

E.g.

(gdb) bt

#0 0x00007fd5ffb4e8f3 in vfprintf () from /lib/x86_64-linux-gnu/libc.so.6

#1 0x00007fd5ffc0d39d in __vfprintf_chk () from /lib/x86_64-linux-gnu/libc.so.6

#2 0x000000000042d74f in vprintf (__ap=0x7fd344a27cf8, __fmt=0x52534d "VI: %s registering [%s] [%d]\n")

at /usr/include/x86_64-linux-gnu/bits/stdio2.h:117

#3 ippe_debug_print (debug=debug@entry=0, fmt=fmt@entry=0x52534d "VI: %s registering [%s] [%d]\n")

at /home/itrinegy/Development/ine-dpdk/ipperte.c:339

#4 0x0000000000428edd in ippe_register_port (

port_name=0x7fd34d368120 <error: Cannot access memory at address 0x7fd34d368120>, input=input@entry=0,

output=output@entry=1) at /home/itrinegy/Development/ine-dpdk/dpdk.c:3311

#5 0x00007fd5fdbfadd3 in __class_Symmetric_Routing_UserInput (inputParameter=0x7fd344a28140,

__this_ptr=0x7fd598db2da0, __td=<optimized out>) at /tmp/Default.c:13627

#6 __class_Symmetric_Routing__ippe_input (__td=<optimized out>, __this_ptr=0x7fd598db2da0, argc=<optimized out>,

argv=<optimized out>) at /tmp/Default.c:13266

#7 0x00000000004309bb in ippe_process_proc (module_id=module_id@entry=0,

proc=proc@entry=0x7fd31c000aed "Symmetric_Routing", index=60, id=<optimized out>, args=0x7fd31d3c1fd0,

args_len=4, __td=__td@entry=0x7fd344a28500, username=username@entry=0x7fd325398ec0 "admin")

at /home/itrinegy/Development/ine-dpdk/ipperte.c:7789

#8 0x0000000000448b7e in update_kernel (args=args@entry=0x7fd31d3b8060, num_args=<optimized out>,

cl_ret=cl_ret@entry=0x7fd31c9c4920 "", UNUSED_fd=UNUSED_fd@entry=43, close_fd=close_fd@entry=0x7fd344a29dd0)

at /home/itrinegy/Development/ine-dpdk/ipperte.c:6278

#9 0x000000000044a005 in interpret_commands (fd=fd@entry=43, buf=buf@entry=0x7fd31c0008c0 "--sessionId",

close_fd=close_fd@entry=0x7fd344a29dd0) at /home/itrinegy/Development/ine-dpdk/ipperte.c:1972

#10 0x000000000044a187 in read_data (data=<optimized out>) at /home/itrinegy/Development/ine-dpdk/ipperte.c:1921

#11 0x00007fd5ffed1182 in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0

#12 0x00007fd5ffbfdfbd in clone () from /lib/x86_64-linux-gnu/libc.so.6

(gdb)

This helps us to see who the caller was and coming up the stack we can see that the first routine we’re responsible for is in stack frame 3:

#3 ippe_debug_print (debug=debug@entry=0, fmt=fmt@entry=0x52534d "VI: %s registering [%s] [%d]\n")

at /home/itrinegy/Development/ine-dpdk/ipperte.c:339

Where we see that ippe_debug_print caused the crash and the offending line was 339 of /home/itrinegy/Development/ine-dpdk/ipperte.c

Of course if we’re looking at pepl code then we have to continue to frame 5:

#5 0x00007fd5fdbfadd3 in __class_Symmetric_Routing_UserInput (inputParameter=0x7fd344a28140,

__this_ptr=0x7fd598db2da0, __td=<optimized out>) at /tmp/Default.c:13627

Where we see that ippe_debug_print caused the crash and the offending line was 13627 of /tmp/Default.c - which is the temporary file created by ./peplc default_Enterprise.pepl - the Default comes from the module name at the top of this pepl source file

Full Stack Traces (bt full)

Sometimes the brief backtrace just doesn’t offer enough and we want a full backtrace

Command:

(gdb) bt full

Or

(gdb) backtrace full

E.g.

Core was generated by `./ipperte'.

Program terminated with signal SIGSEGV, Segmentation fault.

#0 0x00007fd5ffb4e8f3 in vfprintf () from /lib/x86_64-linux-gnu/libc.so.6

(gdb) bt full

#0 0x00007fd5ffb4e8f3 in vfprintf () from /lib/x86_64-linux-gnu/libc.so.6

No symbol table info available.

#1 0x00007fd5ffc0d39d in __vfprintf_chk () from /lib/x86_64-linux-gnu/libc.so.6

No symbol table info available.

#2 0x000000000042d74f in vprintf (__ap=0x7fd344a27cf8, __fmt=0x52534d "VI: %s registering [%s] [%d]\n")

at /usr/include/x86_64-linux-gnu/bits/stdio2.h:117

No locals.

#3 ippe_debug_print (debug=debug@entry=0, fmt=fmt@entry=0x52534d "VI: %s registering [%s] [%d]\n")

at /home/itrinegy/Development/ine-dpdk/ipperte.c:339

args = {{gp_offset = 24, fp_offset = 48, overflow_arg_area = 0x7fd344a27e90,

reg_save_area = 0x7fd344a27dc0}}

t = 1465210889

s = "2016-06-06 11:01:29\000\000\000\000\000qy\273\377\325\177\000\000`~\242D\323\177\000\000\340\017\354\377\325\177\000\000\000\000\000\000\000\000\000\000\tXUW\000\000\000\000\340v+M\323\177\000\000 ~\242D\323\177\000\000 ~\242D\323\177\000\000\020~\242D\323\177\000\000\001\000\000"

tm = {tm_sec = 29, tm_min = 1, tm_hour = 11, tm_mday = 6, tm_mon = 5, tm_year = 116, tm_wday = 1,

tm_yday = 157, tm_isdst = 0, tm_gmtoff = 0, tm_zone = 0x7fd5ffc81d50 "GMT"}

#4 0x0000000000428edd in ippe_register_port (

port_name=0x7fd34d368120 <error: Cannot access memory at address 0x7fd34d368120>, input=input@entry=0,

output=output@entry=1) at /home/itrinegy/Development/ine-dpdk/dpdk.c:3311

core = 0x7fd574041e80

port = -1

i = <optimized out>

port_list = 0x0

port_count = 0

#5 0x00007fd5fdbfadd3 in __class_Symmetric_Routing_UserInput (inputParameter=0x7fd344a28140,

__this_ptr=0x7fd598db2da0, __td=<optimized out>) at /tmp/Default.c:13627

temp_list = {size = 0, used = 0, last_accessed_index = 0, modified = 0 '\000', ref_required = 1 '\001',

type = 0 '\000', head = 0x0, tail = 0x0, used_tail = 0x0, last_accessed = 0x0, reset = 0x0,

block_head = 0x0, block_tail = 0x0, block_size = 16, number_of_blocks = 0}

number_in_range = <optimized out>

a = 0x0

__Routes_ptr = <optimized out>

__VLAN_Id_ptr = 0x0

n = <optimized out>

r = <optimized out>

__this = 0x7fd598db2da0

number_in_list = <optimized out>

temp_range = {size = 0, used = 0, last_accessed_index = 0, modified = 0 '\000', ref_required = 1 '\001',

type = 0 '\000', head = 0x0, tail = 0x0, used_tail = 0x0, last_accessed = 0x0, reset = 0x0,

block_head = 0x0, block_tail = 0x0, block_size = 16, number_of_blocks = 0}

__parameter_ptr = <optimized out>

__sb = {which = 1, {fptr = 0x7fd34d368120,

buffer = 0x7fd34d368120 <error: Cannot access memory at address 0x7fd34d368120>}, size = 2048,

used = 6, next = 0x0}

__temp_list_ptr = <optimized out>

__IPAddress_ptr = 0x0

__IPPort_ptr = 0x0

Indexes = {size = 0, used = 0, last_accessed_index = 0, modified = 0 '\000', ref_required = 0 '\000',

type = 0 '\000', head = 0x0, tail = 0x0, used_tail = 0x0, last_accessed = 0x0, reset = 0x0,

block_head = 0x0, block_tail = 0x0, block_size = 16, number_of_blocks = 0}

p = <optimized out>

list_item = <optimized out>

#6 __class_Symmetric_Routing__ippe_input (__td=<optimized out>, __this_ptr=0x7fd598db2da0, argc=<optimized out>,

argv=<optimized out>) at /tmp/Default.c:13266

__this = <optimized out>

i = <optimized out>

indexes = {size = 0, used = 0, last_accessed_index = 0, modified = 0 '\000', ref_required = 0 '\000',

type = 0 '\000', head = 0x0, tail = 0x0, used_tail = 0x0, last_accessed = 0x0, reset = 0x0,

block_head = 0x0, block_tail = 0x0, block_size = 16, number_of_blocks = 0}

args = {vi = 0x0, emulation = 0x0, vi_data = 0x0, emulation_data = 0x0, global_data = 0x0,

parameter_size_ = 2, parameter = {size = 2, used = 2, last_accessed_index = 0, modified = 1 '\001',

ref_required = 1 '\001', type = 10 '\n', head = 0x7fd58f79f520, tail = 0x7fd58f79f538,

used_tail = 0x7fd58f79f538, last_accessed = 0x0,

reset = 0x44a2c0 <__reset_object___ippe_module_input_data_params>, block_head = 0x7fd58f79f520,

block_tail = 0x7fd58f79f520, block_size = 2, number_of_blocks = 1}}

#7 0x00000000004309bb in ippe_process_proc (module_id=module_id@entry=0,

proc=proc@entry=0x7fd31c000aed "Symmetric_Routing", index=60, id=<optimized out>, args=0x7fd31d3c1fd0,

args_len=4, __td=__td@entry=0x7fd344a28500, username=username@entry=0x7fd325398ec0 "admin")

---Type <return> to continue, or q <return> to quit---

at /home/itrinegy/Development/ine-dpdk/ipperte.c:7789

core = 0x7fd574041e80

m = <optimized out>

i = 0x33b2570

pos = 0x7fd3505045c0

prev = <optimized out>

remove = 0x0

count = <optimized out>

vi = 0x7fd573e5d900

new = 0

current_proc = 0xa58500 <ippe_global_proc>

current_output = 0x7fd5ffec3400 <_IO_2_1_stdout_>

#8 0x0000000000448b7e in update_kernel (args=args@entry=0x7fd31d3b8060, num_args=<optimized out>,

cl_ret=cl_ret@entry=0x7fd31c9c4920 "", UNUSED_fd=UNUSED_fd@entry=43, close_fd=close_fd@entry=0x7fd344a29dd0)

at /home/itrinegy/Development/ine-dpdk/ipperte.c:6278

__td = {ippe_module_error_code = 0, ippe_module_error_message = 0x0, memory = 0x0}

object = 0x7fd31c000aed "Symmetric_Routing"

module_id = 0

m = 0x7fd31d3c1b30

tmpm = 0x7fd31d3c2000

index = <optimized out>

i = <optimized out>

pid = 10059

vi = 0x7fd31d3bb1b0

start = <optimized out>

stop = <optimized out>

stop_all = <optimized out>

getif = <optimized out>

get_services = <optimized out>

get_emulations = <optimized out>

get_vi_settings = <optimized out>

get_vis_for_emulation = <optimized out>

get_version = <optimized out>

license_status = <optimized out>

get_port_settings = <optimized out>

get_service_settings = <optimized out>

save_running_as_default = <optimized out>

user_set = <optimized out>

update_port = <optimized out>

port_remove = <optimized out>

update_service = <optimized out>

ret = 0

no_print = <optimized out>

error = 0

point = 0

emulation_id = <optimized out>

emulation_name = <optimized out>

notes = <optimized out>

noop = <optimized out>

get_modules = <optimized out>

get_port_stats = <optimized out>

get_module_interfaces = <optimized out>

get_module_object = <optimized out>

core = 0x7fd574041e80

test = <optimized out>

port_name = <optimized out>

license_element = <optimized out>

test_result = 0

service_remove = <optimized out>

dump_memory = <optimized out>

user = <optimized out>

tmp_user = {sid = '\000' <repeats 63 times>, username = '\000' <repeats 63 times>, login = 0,

last_accessed = 0, expire_time = 0, prev = 0x0, next = 0x0}

admin_user = <optimized out>

uc = <optimized out>

save_user_emulation = <optimized out>

delete_user_emu = <optimized out>

run_emulation = <optimized out>

list_emulations = <optimized out>

---Type <return> to continue, or q <return> to quit---

create_folder = <optimized out>

delete_folder = <optimized out>

get_file = <optimized out>

upload_file = <optimized out>

rename_file = <optimized out>

tail_file = <optimized out>

share_file = <optimized out>

unshare_file = <optimized out>

copy_file = <optimized out>

list_file_shares = <optimized out>

capture_packets = <optimized out>

stop_capture_packets = <optimized out>

configure_port = <optimized out>

get_number_vis_for_emulation = <optimized out>

delete_default_emulation = <optimized out>

verify_emulation = <optimized out>

#9 0x000000000044a005 in interpret_commands (fd=fd@entry=43, buf=buf@entry=0x7fd31c0008c0 "--sessionId",

close_fd=close_fd@entry=0x7fd344a29dd0) at /home/itrinegy/Development/ine-dpdk/ipperte.c:1972

result = 28

point = 0

args = 0x7fd31d3b8060

i = <optimized out>

pid = 10059

ret = 0x7fd31c9c4920 ""

j = 0

#10 0x000000000044a187 in read_data (data=<optimized out>) at /home/itrinegy/Development/ine-dpdk/ipperte.c:1921

fd = 43

bytes = <optimized out>

buf = 0x7fd31c0008c0 "--sessionId"

ptr = 0x7fd31c000cbb ""

total_read = 1019

i = 0

used = <optimized out>

pid = 10059

close_fd = 0

#11 0x00007fd5ffed1182 in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0

No symbol table info available.

#12 0x00007fd5ffbfdfbd in clone () from /lib/x86_64-linux-gnu/libc.so.6

No symbol table info available.

(gdb)

This gives much more information about the parameter passed etc, unfortunately in the case of IPPERTE/PEPL much is optimised out

Equating c names to pepl names

C Function names to PEPL Class and Procs

There’s no exact way of doing this for scalar variables the pepl and c variable names are identical for example in frame 5 of bt full above:

#5 0x00007fd5fdbfadd3 in __class_Symmetric_Routing_UserInput (inputParameter=0x7fd344a28140,

__this_ptr=0x7fd598db2da0, __td=<optimized out>) at /tmp/Default.c:13627

temp_list = {size = 0, used = 0, last_accessed_index = 0, modified = 0 '\000', ref_required = 1 '\001',

type = 0 '\000', head = 0x0, tail = 0x0, used_tail = 0x0, last_accessed = 0x0, reset = 0x0,

block_head = 0x0, block_tail = 0x0, block_size = 16, number_of_blocks = 0}

number_in_range = <optimized out>

a = 0x0

__Routes_ptr = <optimized out>

__VLAN_Id_ptr = 0x0

n = <optimized out>

r = <optimized out>

__this = 0x7fd598db2da0

number_in_list = <optimized out>

temp_range = {size = 0, used = 0, last_accessed_index = 0, modified = 0 '\000', ref_required = 1 '\001',

type = 0 '\000', head = 0x0, tail = 0x0, used_tail = 0x0, last_accessed = 0x0, reset = 0x0,

block_head = 0x0, block_tail = 0x0, block_size = 16, number_of_blocks = 0}

__parameter_ptr = <optimized out>

__sb = {which = 1, {fptr = 0x7fd34d368120,

buffer = 0x7fd34d368120 <error: Cannot access memory at address 0x7fd34d368120>}, size = 2048,

used = 6, next = 0x0}

__temp_list_ptr = <optimized out>

__IPAddress_ptr = 0x0

__IPPort_ptr = 0x0

Indexes = {size = 0, used = 0, last_accessed_index = 0, modified = 0 '\000', ref_required = 0 '\000',

type = 0 '\000', head = 0x0, tail = 0x0, used_tail = 0x0, last_accessed = 0x0, reset = 0x0,

block_head = 0x0, block_tail = 0x0, block_size = 16, number_of_blocks = 0}

p = <optimized out>

list_item = <optimized out>

We see the c function name: __class_Symmetric_Routing_UserInput

This equates to the pepl class Symmetric_Routing and the Proc (method) within it called UserInput

So now we clearly know what pepl proc was running at the time of the problem. Stack frame 4 tells us that the pepl had called: ippe_register_port

Internal Function names

So from the above we see that the function ippe_register_port was called, but in our PEPL source code there is no such call. Of course we might guess that’s it’ll be related to port registration but there is a way to be more certain:

Search the file ~/ine/import/system/pepl-defs.pepl for ippe_register_port (using vi or vim to get the context. This mostly, but not always delivers results. In this case we find the block:

Cdecl Proc registerPort

Input String p

Input TrueFalse input_required

Input TrueFalse output_required

Return Number

EndProc

Calias Proc registerPort ippe_register_port

Which tells us that the c function ippe_register_port is aliased to the PEPL proc registerPort and so that’s what we’re looking for in our PEPL source code. It also tells us that there are 3 parameters to this function and from stack frame 4 we see:

port_name=0x7fd34d368120 <error: Cannot access memory at address 0x7fd34d368120>, input=input@entry=0,

output=output@entry=1)

So we don’t know what the first (port_name) parameter is but we do know that the next two parameters are 0 and 1 respectively and so our call in PEPL will look something like: registerPort(port_name,0,1)

In fact we find 3 instances of registerPort in UserInput in class Symmetric_Routing:

Port_In_Id = registerPort(Port_In, True, False)

if Port_In_Id > IPPE_PORT_ERROR_START

if Port_In_Id = IPPE_PORT_UNSPECIFIED

# It's allowed

elif Port_In_Id = IPPE_PORT_REGISTERED

Error IPPE_PORT_REGISTERED "Object " + currentVI.name + ": " + "Input port (" + Port_In + ") already in use"

elif Port_In_Id = IPPE_INVALID_PORT

Error IPPE_INVALID_PORT "Object " + currentVI.name + ": " + "No such port (" + Port_In + ") available"

else

Error 1 "Object " + currentVI.name + ": " + "Unknown port error: " + Port_In_Id

endif

Port_Out_Id = registerPort(Port_Out, False, True)

if Port_Out_Id > IPPE_PORT_ERROR_START

if Port_Out_Id = IPPE_PORT_UNSPECIFIED

# It's allowed

elif Port_Out_Id = IPPE_PORT_REGISTERED

Error IPPE_PORT_REGISTERED "Object " + currentVI.name + ": " + "Output port (" + Port_Out + ") already in use"

elif Port_Out_Id = IPPE_INVALID_PORT

Error IPPE_INVALID_PORT "Object " + currentVI.name + ": " + "No such port (" + Port_Out + ") available"

else

Error 1 "Object " + currentVI.name + ": " + "Unknown port error: " + Port_Out_Id

endif

# start assuming no default route

Default_Route_Id = IPPE_INVALID_PORT # This is 64 bit -1 in an unsigned int64

Route_to_Default = False

log ""

log " Processing Routes for VI: " + currentVI.name

log ""

# now check the array

forall Routes r

r.Port_In_Id = toPortId(r.Port_In)

# the "error" IPPE_PORT_UNSPECIFIED will be treated as valid lower down

r.Port_Out_Id = registerPort(r.Port_Out, False, True)

…etc.

The first of these is eliminated because its parameters are the wrong way round: Internally False = 0 and True = 1 in PEPL

The one in the forall loop was eliminated by looking at the ippe.log file and noting which VI was being updated. It was a line object and that had an empty routing table.

That left the second instance:

Port_Out_Id = registerPort(Port_Out, False, True)

Variable names

Variables in PEPL generally match closely to their C names. Here are some variables defined half way into the PEPL proc UserInput in the class we are using as an example Symmetric_Routing

string array temp_list

number number_in_list

string array temp_range

number number_in_range

number n

number i

And looking in the file /tmp/Default.c which was cross compiled from ~/ine/ine-classes/default_Enterprise.pepl, we see:

uint64_t __class_Symmetric_Routing_UserInput(struct __ippe_thread_data *__td, void *__this_ptr, struct __ippe_module_input_data *inputParameter) {

struct __ippe_class_Symmetric_Routing *__this = (struct __ippe_class_Symmetric_Routing *)__this_ptr;

struct __ippe_module_input_data_params *__parameter_ptr = NULL;

struct smart_buffer __sb = {1, {NULL}, 0, 0, NULL};

struct Symmetric_Link_route *__Routes_ptr = NULL;

char *__temp_list_ptr;

struct IPV4Address_Range *__IPAddress_ptr = NULL;

struct __number_range *__IPPort_ptr = NULL;

struct __number_range *__VLAN_Id_ptr = NULL;

struct ippe_list_array_head Indexes;

struct ippe_list_array_head temp_list;

uint64_t number_in_list = 0;

struct ippe_list_array_head temp_range;

uint64_t number_in_range = 0;

uint64_t n = 0;

struct ippe_list_array *p = NULL;

struct ippe_list_array *r = NULL;

struct ippe_list_array *list_item = NULL;

struct ippe_list_array *a = NULL;

Notice how the names are preserved in the C, though arrays become: struct ippe_list_array_head types. Also notice that even though we declared them half way up a proc they are moved into a struct which has all the variables at the top of the corresponding c function.

For completeness it worth noting that:

What happened to the variable declared as: Number i - it is not in the c struct? The answer to this is that it was never used and so PEPL optimised it out.

Displaying Variable Contents

While bt full will show you the values of variables in the stack frames - which is good sometimes we want to see the contents of memory, where available:

For this we use print e.g.

(gdb) print m->module[0]

Cannot access memory at address 0x0

(gdb) print m->module

$1 = (char **) 0x0

(gdb) print *m

$2 = {module = 0x0, result = -1, next = 0x7f516dce8da0}

(gdb) print num_args

$3 = <optimized out>

(gdb) print args[0]

$4 = 0x7f516c000a00 "--sessionId"

(gdb) print args[1]

$5 = 0x7f516c000a0c "a1d03fc1a8720f8047c8236f91815321"

(gdb) print args[2]

$6 = 0x7f516c000a2d "--id"

(gdb) print args[3]

$7 = 0x7f516c000a32 "3584"

(gdb) print args[4]

$8 = 0x7f516c000a37 "--name"

Notice that pointers -> are followed and we can look at the contents of a pointed to variable with *. Essentially c syntax. Also array variables are referenced as in c enclosed by [ ]

You can also hex dump an area of memory using x/<bytes>bx <pointer>:

(gdb) x/512bx args

0x7f516dc981c0: 0x00 0x0a 0x00 0x6c 0x51 0x7f 0x00 0x00

0x7f516dc981c8: 0x0c 0x0a 0x00 0x6c 0x51 0x7f 0x00 0x00

0x7f516dc981d0: 0x2d 0x0a 0x00 0x6c 0x51 0x7f 0x00 0x00

0x7f516dc981d8: 0x32 0x0a 0x00 0x6c 0x51 0x7f 0x00 0x00

0x7f516dc981e0: 0x37 0x0a 0x00 0x6c 0x51 0x7f 0x00 0x00

0x7f516dc981e8: 0x3e 0x0a 0x00 0x6c 0x51 0x7f 0x00 0x00

0x7f516dc981f0: 0x69 0x0a 0x00 0x6c 0x51 0x7f 0x00 0x00

0x7f516dc981f8: 0x76 0x0a 0x00 0x6c 0x51 0x7f 0x00 0x00

0x7f516dc98200: 0xa7 0x0a 0x00 0x6c 0x51 0x7f 0x00 0x00

0x7f516dc98208: 0xb4 0x0a 0x00 0x6c 0x51 0x7f 0x00 0x00

0x7f516dc98210: 0xcf 0x0a 0x00 0x6c 0x51 0x7f 0x00 0x00

0x7f516dc98218: 0xdc 0x0a 0x00 0x6c 0x51 0x7f 0x00 0x00

0x7f516dc98220: 0x0a 0x0b 0x00 0x6c 0x51 0x7f 0x00 0x00

0x7f516dc98228: 0x17 0x0b 0x00 0x6c 0x51 0x7f 0x00 0x00

0x7f516dc98230: 0x5c 0x0b 0x00 0x6c 0x51 0x7f 0x00 0x00

0x7f516dc98238: 0x69 0x0b 0x00 0x6c 0x51 0x7f 0x00 0x00

0x7f516dc98240: 0x89 0x0b 0x00 0x6c 0x51 0x7f 0x00 0x00

0x7f516dc98248: 0x96 0x0b 0x00 0x6c 0x51 0x7f 0x00 0x00

0x7f516dc98250: 0xe5 0x0b 0x00 0x6c 0x51 0x7f 0x00 0x00

0x7f516dc98258: 0xf2 0x0b 0x00 0x6c 0x51 0x7f 0x00 0x00

0x7f516dc98260: 0x38 0x0c 0x00 0x6c 0x51 0x7f 0x00 0x00

0x7f516dc98268: 0x45 0x0c 0x00 0x6c 0x51 0x7f 0x00 0x00

0x7f516dc98270: 0xb7 0x0c 0x00 0x6c 0x51 0x7f 0x00 0x00

0x7f516dc98278: 0xc4 0x0c 0x00 0x6c 0x51 0x7f 0x00 0x00

0x7f516dc98280: 0x1e 0x0d 0x00 0x6c 0x51 0x7f 0x00 0x00

0x7f516dc98288: 0x2b 0x0d 0x00 0x6c 0x51 0x7f 0x00 0x00

0x7f516dc98290: 0xb7 0x0d 0x00 0x6c 0x51 0x7f 0x00 0x00

0x7f516dc98298: 0xc4 0x0d 0x00 0x6c 0x51 0x7f 0x00 0x00

0x7f516dc982a0: 0xf0 0x00 0x00 0x00 0x00 0x00 0x00 0x00

0x7f516dc982a8: 0x45 0x00 0x00 0x00 0x00 0x00 0x00 0x00

0x7f516dc982b0: 0x97 0x0b 0x00 0x6c 0x51 0x7f 0x00 0x00

0x7f516dc982b8: 0xac 0x0b 0x00 0x6c 0x51 0x7f 0x00 0x00

0x7f516dc982c0: 0xaf 0x0b 0x00 0x6c 0x51 0x7f 0x00 0x00

0x7f516dc982c8: 0xb9 0x0b 0x00 0x6c 0x51 0x7f 0x00 0x00

0x7f516dc982d0: 0xbb 0x0b 0x00 0x6c 0x51 0x7f 0x00 0x00

0x7f516dc982d8: 0xd5 0x0b 0x00 0x6c 0x51 0x7f 0x00 0x00

0x7f516dc982e0: 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00

0x7f516dc982e8: 0x45 0x00 0x00 0x00 0x00 0x00 0x00 0x00

0x7f516dc982f0: 0xf3 0x0b 0x00 0x6c 0x51 0x7f 0x00 0x00

0x7f516dc982f8: 0x0b 0x0c 0x00 0x6c 0x51 0x7f 0x00 0x00

0x7f516dc98300: 0x0e 0x0c 0x00 0x6c 0x51 0x7f 0x00 0x00

0x7f516dc98308: 0x16 0x0c 0x00 0x6c 0x51 0x7f 0x00 0x00

0x7f516dc98310: 0x1e 0x0c 0x00 0x6c 0x51 0x7f 0x00 0x00

0x7f516dc98318: 0x27 0x0c 0x00 0x6c 0x51 0x7f 0x00 0x00

0x7f516dc98320: 0xff 0xff 0xff 0xff 0x00 0x00 0x00 0x00

0x7f516dc98328: 0x25 0x00 0x00 0x00 0x00 0x00 0x00 0x00

0x7f516dc98330: 0xab 0x0a 0x00 0x6c 0x51 0x7f 0x00 0x00

0x7f516dc98338: 0xc2 0x0a 0x00 0x6c 0x51 0x7f 0x00 0x00

0x7f516dc98340: 0x20 0x00 0x00 0x00 0x00 0x00 0x00 0x00

0x7f516dc98348: 0x65 0x00 0x00 0x00 0x00 0x00 0x00 0x00

0x7f516dc98350: 0x3d 0x0c 0x00 0x6c 0x51 0x7f 0x00 0x00

0x7f516dc98358: 0x5f 0x0c 0x00 0x6c 0x51 0x7f 0x00 0x00

0x7f516dc98360: 0x62 0x0c 0x00 0x6c 0x51 0x7f 0x00 0x00

0x7f516dc98368: 0x74 0x0c 0x00 0x6c 0x51 0x7f 0x00 0x00

0x7f516dc98370: 0x7d 0x0c 0x00 0x6c 0x51 0x7f 0x00 0x00

0x7f516dc98378: 0x8e 0x0c 0x00 0x6c 0x51 0x7f 0x00 0x00

0x7f516dc98380: 0x90 0x0c 0x00 0x6c 0x51 0x7f 0x00 0x00

0x7f516dc98388: 0x9d 0x0c 0x00 0x6c 0x51 0x7f 0x00 0x00

0x7f516dc98390: 0x9f 0x0c 0x00 0x6c 0x51 0x7f 0x00 0x00

0x7f516dc98398: 0xac 0x0c 0x00 0x6c 0x51 0x7f 0x00 0x00

0x7f516dc983a0: 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00

0x7f516dc983a8: 0x25 0x00 0x00 0x00 0x00 0x00 0x00 0x00

0x7f516dc983b0: 0xb4 0x0a 0x00 0x6c 0x51 0x7f 0x00 0x00

0x7f516dc983b8: 0xcb 0x0a 0x00 0x6c 0x51 0x7f 0x00 0x00

For a character dump use x/<bytes>c <pointer> :

(gdb) x/512c buf

0x7f516c000a00: 45 '-' 45 '-' 115 's' 101 'e' 115 's' 115 's' 105 'i' 111 'o'

0x7f516c000a08: 110 'n' 73 'I' 100 'd' 0 '\000' 97 'a' 49 '1' 100 'd' 48 '0'

0x7f516c000a10: 51 '3' 102 'f' 99 'c' 49 '1' 97 'a' 56 '8' 55 '7' 50 '2'

0x7f516c000a18: 48 '0' 102 'f' 56 '8' 48 '0' 52 '4' 55 '7' 99 'c' 56 '8'

0x7f516c000a20: 50 '2' 51 '3' 54 '6' 102 'f' 57 '9' 49 '1' 56 '8' 49 '1'

0x7f516c000a28: 53 '5' 51 '3' 50 '2' 49 '1' 0 '\000' 45 '-' 45 '-' 105 'i'

0x7f516c000a30: 100 'd' 0 '\000' 51 '3' 53 '5' 56 '8' 52 '4' 0 '\000' 45 '-'

0x7f516c000a38: 45 '-' 110 'n' 97 'a' 109 'm' 101 'e' 0 '\000' 70 'F' 117 'u'

0x7f516c000a40: 108 'l' 108 'l' 32 ' ' 68 'D' 117 'u' 112 'p' 108 'l' 101 'e'

0x7f516c000a48: 120 'x' 54 '6' 58 ':' 32 ' ' 82 'R' 111 'o' 117 'u' 116 't'

0x7f516c000a50: 101 'e' 114 'r' 49 '1' 32 ' ' 45 '-' 45 '-' 62 '>' 32 ' '

0x7f516c000a58: 71 'G' 97 'a' 109 'm' 101 'e' 32 ' ' 67 'C' 111 'o' 110 'n'

0x7f516c000a60: 116 't' 114 'r' 111 'o' 108 'l' 108 'l' 101 'e' 114 'r' 53 '5'

0x7f516c000a68: 0 '\000' 45 '-' 45 '-' 112 'p' 114 'r' 111 'o' 99 'c' 77 'M'

0x7f516c000a70: 111 'o' 100 'd' 117 'u' 108 'l' 101 'e' 0 '\000' 68 'D' 101 'e'

0x7f516c000a78: 102 'f' 97 'a' 117 'u' 108 'l' 116 't' 0 '\000' 68 'D' 101 'e'

0x7f516c000a80: 98 'b' 117 'u' 103 'g' 0 '\000' 49 '1' 48 '0' 0 '\000' 68 'D'

0x7f516c000a88: 117 'u' 109 'm' 112 'p' 95 '_' 80 'P' 97 'a' 99 'c' 107 'k'

0x7f516c000a90: 101 'e' 116 't' 0 '\000' 48 '0' 0 '\000' 66 'B' 121 'y' 116 't'

0x7f516c000a98: 101 'e' 115 's' 95 '_' 116 't' 111 'o' 95 '_' 68 'D' 117 'u'

0x7f516c000aa0: 109 'm' 112 'p' 0 '\000' 56 '8' 48 '0' 0 '\000' 0 '\000' 45 '-'

0x7f516c000aa8: 45 '-' 112 'p' 114 'r' 111 'o' 99 'c' 77 'M' 111 'o' 100 'd'

0x7f516c000ab0: 117 'u' 108 'l' 101 'e' 0 '\000' 68 'D' 101 'e' 102 'f' 97 'a'

0x7f516c000ab8: 117 'u' 108 'l' 116 't' 0 '\000' 71 'G' 101 'e' 110 'n' 101 'e'

0x7f516c000ac0: 114 'r' 105 'i' 99 'c' 95 '_' 70 'F' 105 'i' 108 'l' 116 't'

0x7f516c000ac8: 101 'e' 114 'r' 0 '\000' 50 '2' 48 '0' 0 '\000' 0 '\000' 45 '-'

0x7f516c000ad0: 45 '-' 112 'p' 114 'r' 111 'o' 99 'c' 77 'M' 111 'o' 100 'd'

0x7f516c000ad8: 117 'u' 108 'l' 101 'e' 0 '\000' 68 'D' 101 'e' 102 'f' 97 'a'

0x7f516c000ae0: 117 'u' 108 'l' 116 't' 0 '\000' 80 'P' 97 'a' 99 'c' 107 'k'

0x7f516c000ae8: 101 'e' 116 't' 95 '_' 69 'E' 114 'r' 114 'r' 111 'o' 114 'r'

0x7f516c000af0: 95 '_' 49 '1' 95 '_' 105 'i' 110 'n' 95 '_' 120 'x' 95 '_'

0x7f516c000af8: 66 'B' 105 'i' 116 't' 115 's' 0 '\000' 51 '3' 48 '0' 0 '\000'

0x7f516c000b00: 76 'L' 111 'o' 115 's' 115 's' 95 '_' 120 'x' 0 '\000' 48 '0'

0x7f516c000b08: 0 '\000' 0 '\000' 45 '-' 45 '-' 112 'p' 114 'r' 111 'o' 99 'c'

0x7f516c000b10: 77 'M' 111 'o' 100 'd' 117 'u' 108 'l' 101 'e' 0 '\000' 68 'D'

0x7f516c000b18: 101 'e' 102 'f' 97 'a' 117 'u' 108 'l' 116 't' 0 '\000' 82 'R'

0x7f516c000b20: 97 'a' 110 'n' 100 'd' 111 'o' 109 'm' 95 '_' 80 'P' 97 'a'

0x7f516c000b28: 99 'c' 107 'k' 101 'e' 116 't' 95 '_' 67 'C' 111 'o' 114 'r'

0x7f516c000b30: 114 'r' 117 'u' 112 'p' 116 't' 0 '\000' 52 '4' 48 '0' 0 '\000'

0x7f516c000b38: 80 'P' 97 'a' 99 'c' 107 'k' 101 'e' 116 't' 95 '_' 67 'C'

0x7f516c000b40: 111 'o' 114 'r' 114 'r' 117 'u' 112 'p' 116 't' 105 'i' 111 'o'

0x7f516c000b48: 110 'n' 95 '_' 80 'P' 101 'e' 114 'r' 99 'c' 101 'e' 110 'n'

0x7f516c000b50: 116 't' 0 '\000' 48 '0' 46 '.' 48 '0' 48 '0' 48 '0' 48 '0'

0x7f516c000b58: 48 '0' 48 '0' 0 '\000' 0 '\000' 45 '-' 45 '-' 112 'p' 114 'r'

0x7f516c000b60: 111 'o' 99 'c' 77 'M' 111 'o' 100 'd' 117 'u' 108 'l' 101 'e'

0x7f516c000b68: 0 '\000' 68 'D' 101 'e' 102 'f' 97 'a' 117 'u' 108 'l' 116 't'

0x7f516c000b70: 0 '\000' 70 'F' 105 'i' 120 'x' 101 'e' 100 'd' 95 '_' 68 'D'

0x7f516c000b78: 101 'e' 108 'l' 97 'a' 121 'y' 0 '\000' 53 '5' 48 '0' 0 '\000'

0x7f516c000b80: 68 'D' 101 'e' 108 'l' 97 'a' 121 'y' 0 '\000' 48 '0' 0 '\000'

0x7f516c000b88: 0 '\000' 45 '-' 45 '-' 112 'p' 114 'r' 111 'o' 99 'c' 77 'M'

0x7f516c000b90: 111 'o' 100 'd' 117 'u' 108 'l' 101 'e' 0 '\000' 68 'D' 101 'e'

0x7f516c000b98: 102 'f' 97 'a' 117 'u' 108 'l' 116 't' 0 '\000' 70 'F' 114 'r'

0x7f516c000ba0: 97 'a' 103 'g' 109 'm' 101 'e' 110 'n' 116 't' 95 '_' 77 'M'

0x7f516c000ba8: 84 'T' 85 'U' 0 '\000' 53 '5' 53 '5' 0 '\000' 77 'M' 84 'T'

0x7f516c000bb0: 85 'U' 95 '_' 76 'L' 105 'i' 109 'm' 105 'i' 116 't' 0 '\000'

0x7f516c000bb8: 48 '0' 0 '\000' 68 'D' 111 'o' 110 'n' 116 't' 95 '_' 70 'F'

0x7f516c000bc0: 114 'r' 97 'a' 103 'g' 109 'm' 101 'e' 110 'n' 116 't' 95 '_'

0x7f516c000bc8: 70 'F' 108 'l' 97 'a' 103 'g' 95 '_' 79 'O' 112 'p' 116 't'

0x7f516c000bd0: 105 'i' 111 'o' 110 'n' 0 '\000' 70 'F' 114 'r' 97 'a' 103 'g'

0x7f516c000bd8: 109 'm' 101 'e' 110 'n' 116 't' 32 ' ' 65 'A' 110 'n' 121 'y'

0x7f516c000be0: 119 'w' 97 'a' 121 'y' 0 '\000' 0 '\000' 45 '-' 45 '-' 112 'p'

0x7f516c000be8: 114 'r' 111 'o' 99 'c' 77 'M' 111 'o' 100 'd' 117 'u' 108 'l'

0x7f516c000bf0: 101 'e' 0 '\000' 68 'D' 101 'e' 102 'f' 97 'a' 117 'u' 108 'l'

0x7f516c000bf8: 116 't' 0 '\000' 71 'G' 101 'e' 110 'n' 101 'e' 114 'r' 105 'i'

(gdb) x/512c buf