14d689c0f9
Add prof stats mutex stats
2021-01-07 20:39:49 -08:00
9f71b5779b
Output prof stats in stats print
2021-01-07 20:39:49 -08:00
1f1a0231ed
Split macros for initializing stats headers
2021-01-07 20:39:49 -08:00
54f3351f1f
Add mallctl for prof stats fetching
2021-01-07 20:39:49 -08:00
40fa4d29d3
Track per size class internal fragmentation
2021-01-07 20:39:49 -08:00
afa489c3c5
Record request size in prof info
2021-01-07 20:39:49 -08:00
f9bb8dedef
Un-force-inline do_rallocx.
...
The additional overhead of the function-call setup and flags checking is
relatively small, but costs us the replication of the entire realloc pathway in
terms of size.
2021-01-04 14:55:49 -08:00
a9fa2defdb
Add JEMALLOC_COLD, and mark some functions cold.
...
This hints to the compiler that it should care more about space than CPU (among
other things). In cases where the compiler lacks profile-guided information,
this can be a substantial space savings.
For now, we mark the mallctl or atexit driven profiling and stats functions that
take up the most space.
2021-01-04 14:55:49 -08:00
5d8e70ab26
prof_recent: cassert(config_prof) more often.
...
This tells the compiler that these functions are never called, which lets them
be optimized away in builds where profiling is disabled.
2021-01-04 14:55:49 -08:00
83cad746ae
prof_log: cassert(config_prof) in public functions
...
This lets the compiler infer that the code is dead in builds where profiling is
enabled, saving on space there.
2021-01-04 14:55:49 -08:00
526180b76d
Extent.c: Avoid an rtree NULL-check.
...
The edge case in which pages_map returns (void *)PAGE can trigger an incorrect
assertion failure. Avoid it.
2021-01-04 14:50:49 -08:00
b35ac00d58
Do not bump to large size for page aligned request
2020-12-29 17:09:58 -08:00
8a56d6b636
Add last-N mutex stats
2020-12-29 09:44:19 -08:00
22d62d8cbd
Handle ending gap properly for HPA stats
2020-12-18 16:40:57 -08:00
6c5a3a24dd
Omit bin stats rows with no data
2020-12-18 16:40:57 -08:00
ea013d8fa4
Enforce realloc sizing stability
2020-12-18 11:41:52 -08:00
74bd63b203
Optimize stats print using partial name-to-mib
2020-12-18 10:39:58 -08:00
4557c0a67d
Enable ctl on partial mib and partial name
2020-12-18 10:39:58 -08:00
006dd0414e
Add partial name-to-mib functionality
2020-12-18 10:39:58 -08:00
f2e1a5be77
Do not fail on partial ctl path for ctl_nametomib()
...
We do not fail on partial ctl path when the given `mib` array is
shorter than the given name, and we should keep the behavior the
same in the reverse case, which I feel is also the more natural way.
2020-12-18 10:39:58 -08:00
6ab181d2b7
Extract node lookup given mib input
2020-12-18 10:39:58 -08:00
3a627b9674
No need to record all nodes in ctl_lookup()
2020-12-18 10:39:58 -08:00
91e006c4c2
Enable ctl_lookup() to start from arbitrary node
2020-12-18 10:39:58 -08:00
4e3fe218e9
Use posix_madvise to purge pages when available
2020-12-18 10:05:59 -08:00
1e3b8636ff
HPA: Remove unused malloc_conf options.
2020-12-08 12:10:48 -08:00
9522ae41d6
Move n_search outside of assert as reported by static analyzer
2020-12-07 06:49:27 -08:00
a559caf74a
hpdata: Strengthen assertions.
...
Now that we have flat bitmap bit counting functions, we can easily assert that
nfree is always correct. While we're tightening up this code, enforce
consistency on API boundaries as well.
2020-12-07 06:21:08 -08:00
3ed0b4e8a3
HPA: Add an nevictions counter.
...
I.e. the number of times we've purged a hugepage-sized region.
2020-12-07 06:21:08 -08:00
fffcefed33
malloc_conf: Clarify HPA options.
2020-12-07 06:21:08 -08:00
f7cf23aa4d
psset: Relegate alloc/dalloc to test code.
...
This is no longer part of the "core" functionality; we only need the stub
implementations as an end-to-end test of hpdata + psset interactions when
metadata is being modified. Treat them accordingly.
2020-12-07 06:21:08 -08:00
f9299ca572
HPA: Use psset fit/insert/remove.
...
This will let us remove alloc_new and alloc_reuse functions from the psset.
2020-12-07 06:21:08 -08:00
0971e1e4e3
hpdata: Use addr/size instead of begin/npages.
...
This is easier for the users of the hpdata.
2020-12-07 06:21:08 -08:00
5228d869ee
psset: Use fit/insert/remove as basis functions.
...
All other functionality can be implemented in terms of these; doing so (while
retaining the same API) will be convenient for subsequent refactors.
2020-12-07 06:21:08 -08:00
089f8fa442
Move hpdata bitmap logic out of the psset.
2020-12-07 06:21:08 -08:00
ca30b5db2b
Introduce hpdata_t.
...
Using an edata_t both for hugepages and the allocations within those hugepages
was convenient at first, but has outlived its usefulness. Representing
hugepages explicitly, with their own data structure, will make future
development easier.
2020-12-07 06:21:08 -08:00
43af63fff4
HPA: Manage whole hugepages at a time.
...
This redesigns the HPA implementation to allow us to manage hugepages all at
once, locally, without relying on a global fallback.
2020-12-07 06:21:08 -08:00
c1b2a77933
psset: Move in stats.
...
A later change will benefit from having these functions pulled into a
psset-module set of functions.
2020-12-07 06:21:08 -08:00
d0a991d47b
psset: Add insert/remove functions.
...
These will allow us to (for instance) move pageslabs from a psset dedicated to
not-yet-hugeified pages to one dedicated to hugeified ones.
2020-12-07 06:21:08 -08:00
d438296b1f
narenas_ratio: Accept fractional values.
...
With recent scalability improvements to the HPA, we're experimenting with much
lower arena counts; this gets annoying when trying to test across different
hardware configurations using only the narenas setting.
2020-12-04 23:48:19 -08:00
ecd39418ac
Add fxp: A fixed-point math library.
...
This will be used in the next commit to allow non-integer values for
narenas_ratio.
2020-12-04 23:48:19 -08:00
520b75fa2d
utrace support with label based signature.
2020-11-30 11:43:00 -08:00
92e189be8b
Add some comments to the batch allocation logic flow
2020-11-16 20:58:01 -08:00
d96e4525ad
Route batch allocation of small batch size to tcache
2020-11-16 20:58:01 -08:00
566c4a8594
Slight changes to cache bin internal functions
2020-11-16 20:58:01 -08:00
9545c2cd36
Add sample interval to prof last-N dump
2020-11-13 15:33:27 -08:00
cf2549a149
Add a per-arena oversize_threshold.
...
This can let manual arenas trade off memory and CPU the way auto arenas do.
2020-11-13 13:45:35 -08:00
4ca3d91e96
Rename geom_grow -> exp_grow.
...
This was promised in the review of the introduction of geom_grow, but would have
been painful to do there because of the series that introduced it. Now that
those are comitted, renaming is easier.
2020-11-13 13:42:33 -08:00
b4c37a6e81
Rename edata_tree_t -> edata_avail_t.
...
This isn't a tree any more, and it mildly irritates me any time I see it.
2020-11-13 13:42:11 -08:00
95f0a77fde
Detect pthread_getname_np explicitly.
...
At least one libc (musl) defines pthread_setname_np without defining
pthread_getname_np. Detect the presence of each individually, rather than
inferring both must be defined if set is.
2020-11-11 17:31:22 -08:00
589638182a
Use the edata_cache_small_t in the HPA.
2020-11-05 12:34:43 -08:00
Yinan Zhang