dev: discourse findings/suggestions

Yes, probably. As far as I can tell, the sidebar selectors changed during an update a week ago, so the specific one it uses now may not have been available when sb wrote the script. I noticed because my local sidebar overrides broke overnight and I had to rewrite them.

A good change overall though… The new selectors seem a bit nicer than the old ones were.

1 Thank

Nice! Thanks a ton, seems to fix that particular issue.

Right. My custom menus were also partially broken with respect to the custom order I had established, which hastened the adoption of the Discourse custom sidebar method.

1 Thank

ah

well, when an id is not available for matching, a couple of example approaches more specific than positional selection:

// css leveraging `data-section-name` attr
document.querySelector('[data-section-name="community"] .sidebar-section-content');

// or, worst case scenario, an approach like this is always possible:
// xpath for a .sidebar-section-content sibling following a .sidebar-row containing text 'Community'
document.evaluate('//*[contains(@class,"sidebar-row") and contains(.,"Community")]/following-sibling::*[@class="sidebar-section-content"]', document).iterateNext();

just some ideas for future userscripting.

1 Thank

To follow up on this,

Turns out this is for external FAQ pages. (Very weird option.)

I tried switching the content to make /tos contain the BLF Rules and /faq be the Commercial Seller Rules, URLs to but it turns out that there is a hard-coded link from /rules/faq, and I already have a lot of public posts linking to /rules with the intention of it leading to the BLF Rules.

So I switched it all back, but there was a small improvement gleaned:

This definitely should point to the BLF Rules. So I found a way to edit the string translation to include a direct URL pointing where I need instead of a placeholder:

By registering, you agree to the <a href="%{privacy_link}">privacy policy</a> and <a href="/rules">terms of service</a>.

yeah. it can be used that way, but it doesn’t have to be. i have it linking to a topic on the forum. also note there are SiteSetting.tos_url and SiteSetting.privacy_policy_url that can be used to link external resources too.

however, the SiteSetting.faq_url is specifically goofy (yet useful) because it also splits off a route from the guidelines page and activates stuff in the /about template lol so you get an extra section there (5 instead of 4).

it wasn’t documented anywhere to my satisfaction, so i looked at the code and experimented and then documented it here.

if you like, give that a careful read and then my post with specific suggestions will probably make more sense.

not when you use SiteSetting.faq_url :wink:

the end result of my suggestion would work like this.

/guidelines, /rules, /conduct → Commercial Sellers rules

/faqyour FAQ

/tos → BLF Rules (& any extra ToS you may have) – this is automatically featured in the appropriate points in the user registration process

/privacy → privacy policy (and probably featured in user registration)

yeah, that is a problem. aside from the problem of having to update your pages, the distinction between /rules and /tos to mean commercial sellers rules vs general rules is not clear if you care about meaning in urls. that said, the same problem exists with the current setup as well.

updating the public facing posts to reflect the new url is annoying but not insurmountable. you could use the site search to find them, or for more precision, you could use a rails active record query (even using SQL if you want).

you could even use such an approach to automatically fix them all in one go. i could help if you want.

or a completely new option could be to create your own url routes and change the /about template, but i have not explored the plugin system yet, and i don’t know what plugins are capable of yet. and i’d only want to do that with a plugin because i wouldn’t want to make my own patching system for a feature of such low importance.


ps: i’m rewriting the sidebar injector to use mutationobserver because i want it for my site. will share it here when i’m done.

1 Thank

The thing is I don’t really care about having the BLF Discourse FAQs page on the 3 or 4 buttons after /about. Very few users would ever find it or think to look for it there unless I added it to the Community menu (and not hidden under More…). But I guess that setting faq url to /t/new-blf-forum-engine-faqs/217457 kind of works because that way /rules still points to the BLF Rules, which is important for me to maintain because that’s what has the most historical links to it. I’ve had more than enough experience with regexp to know that I really really don’t want to deal with replacing strings across millions of posts. A heads-up for you about that:

  • I heavily tested the rake posts:remap text replacement functionality, and I found it to be totally random about replacing a fixed number of strings followed by a small seemingly random number of strings, and it never replaced all of them.
  • I also extensively tried using the Postgres console directly with the regexp_replace function, but it was also terribly unpredictable because Postgres doesn’t support the most common Perl-Compatible Regular Expressions (PCRE) flavor that I tested my regexp with.
  • Also editing the raw post content would require rebaking all the posts, which would mean I would have to take the forum offline for a few minutes to rescale the VM to have much more RAM and CPU cores, and then let the rake posts:rebake process crunch away on millions of posts for about 48 hours. [2023-02-14] Apologies for the downtime was the result of trying to rebake all the posts with the current VM while online. After that I performed many troubleshooting attempts with a snapshot of the system in another VM, and I even tested different flavours of Linux with all sorts of combinations of swap files and/or zram, but it could not complete the rebake due to memory problems and bizarre CPU hang issues, producing fatal errors ranging all the way from Ruby errors to kernel panics. And that was without any live traffic.

So I can live with this I think:

Nice! Thanks! Looking forward to trying it.

this is the right way of doing it :sunglasses:

it uses promises and MutationObserver.

absent a bug, it should be free of jank. any kind of collapsing, expanding, and refreshing should just “do it” in a performant way.

that took me way too long to write. i’m not a js guy. give it a proofread. maybe i missed a semicolon or something somewhere.

when the sidebar mutates, a promise is created for each link that is resolved when the link’s relevant section is available.

specify injections by section. kebab must be text in kebab case. you also specify which existing element to inject before or after with its kebab’d suffix (sidebar-section-link-SUFFIX), which should just be the name of the link in kebab case anyways.

you can also remove elements. just make removals an empty array if you’re not using that.

edit: i made some changes to the script and made more positioning options. just refer to the example at the top of the script. append: true is probably the positioner you want to use.

2 Thanks

Nice! Really appreciate you sharing that. I’ll be pretty busy these next couple of days, but I hope to give your code a try some time this week.

also there is some logging in there. maybe leave it while playing with it and testing, but you can remove those few console.log lines after.

1 Thank

i actually developed it using tampermonkey (browser extension to inject userscripts into pages), but i haven’t succeeded in getting it to work yet from the discourse admin interface as a theme component yet. hrmmm

will update when i get it working

edit:

oh, i see what the problem is. i developed it using tampermonkey + your site.

my code only fires upon sidebar mutation. i never handled the initial case where the sidebar is present, but no mutations have happened (like when you refresh with the sidebar open).

my code was working without that because your existing sidebar code was mutating the sidebar. lol

1 Thank

ok all fixed.

to test, just paste the whole thing into the head section of a new theme component. append: true is probably the only positioner you need, but i made it pretty flexible.

1 Thank

fwiw, i thought your faq was very helpful, as BLF (post-changeover) was my first in-depth exposure to discourse. i also based my faq on yours.

when i change over, i’ll actually have my FAQ page (a topic) pinned as the banner topic for a month or so with details about the migration and how to get the most out of the new forum. i’ll also leave a FAQ link in the sidebar.

on BLF, it’s kinda had to find though since it’s not in the sidebar, and /about (which prominently links to it now) isn’t in the sidebar, either. i think it doesn’t hurt to prominently display the instruction manual, especially when it’s a nice document :slight_smile:

1 Thank

interesting. thanks for sharing that experience.

i have not personally looked at the posts:remap yet. i actually didn’t know it existed, but for something so finicky as bulk text modification, i’d want to roll my own anyways. and since you say the behavior is goofy, that goes double.

i’ve got some experience with rails now and have written my own rake tasks to do similar things.

i’ve learned it’s almost always a bad idea to deal with the db directly with a rails app. you should, at a minimum, be using methods like .update() on active record objects because it often does validations and may hook other relations to do good things.

i think this specific example is not such a hard one to tackle. food for thought…

if i wanted to bulk-replace /rules to /tos in all posts:

i’d start like this using a postgres regex for efficiency (just for initial identification of posts)

# your_regex is a trivial regex to find posts with links to the rules page
# i would use the cooked column here because it mostly guarantees finding real links
# because links in the cooked version are html tags, so you can match on <a href=...
rules_posts = Post.where('cooked ~ ?', your_regex)

before modifying, i’d hold onto the post ids (maybe even write to a file for safekeeping) so you can figure out rebaking later in another step.

# get an array of all the post ids
matching_post_ids = rules_posts.pluck(:id)

after that, i would iterate over the matching posts and use regex tools in ruby itself (not postgres) to replace the urls in the raw text and update each post like:

rules_posts.each do |p|
  # fix the urls
  old_post_text = p.raw
  # do your string replacements
  # new_post_text = ...some magic
  # update the post
  p.update(raw: new_post_text)
end

when all’s good, i would then rebake just those posts at a leisurely pace. no need to scale up your hardware.

# those ids you savd from before :)
matching_post_ids.sort.reverse.each do |id|
  puts "rebaking post #{id}"
  Post.find(id).rebake!
  sleep(1)
end

that will rebake them in descending chronological order (newest posts first), which i think makes sense. it will also wait a leisurely 1s between posts, which is probably unnecessary. look at matching_post_ids.length and experiment with delay. maybe just try the first 100 posts to start.

btw, you can put your script in a rake task like this:

rules_link_replace.rb

desc 'replace /rules liks to /tos'
task :rules_replace => :environment do
  require 'pry'
  #...
  # btw, you can drop into an interactive ruby env at any point in your code
  # just put this line wherever you want:
  binding.pry # debugging happens here
end

then run the script with a helper like this:

run_rules.sh

#!/usr/bin/env bash

container=app

docker cp ./rules_link_replace.rb "$container:/var/www/discourse/lib/tasks/"
docker exec -it "$container" rake rules_replace
1 Thank

This looks very useful, thanks! You might consider submitting that as a utility for inclusion in Discourse. I like how it will only rebake the modified posts. And what about rebaking the post immediately after its raw text gets modified within the same step of the loop?

nothing wrong with that.

i just assumed that it wouldn’t work right on the first try (and that maybe this would be done on a duplicated dev instance), so no reason to waste the cpu cycles until you’re satisfied with the results of the replacement.

also wasn’t sure how many posts. if it’s only a thousand or something (i mean come on, BLF isn’t exactly a fascist state, how many posts with links to rules could there be) then who cares. doesn’t hurt to rebake while experimenting in that case.

when my migration’s done, i might share my rake tasks, but honestly it’s all such a kludge.

writing general-purpose tools is hard. e.g. apparently posts:remap is ass (as you say), and idk if i could do better.

if anything, i think my code would just serve as a tutorial for things that are possible to do for someone unfamiliar with rails.

PS: after using my sidebar code for a while, it seems pretty bulletproof

I just implemented it here, and I love it! Really well done, clean code, and no irritating 100ms delay before the custom menus get injected.

There’s a minor detail that I’m trying to figure out, for anonymous (not logged in) visitors the menu structure is a bit different. It puts the “Users” link under the “More” menu for logged in users, which is fine, but for anonymous users it shows “Users” at same main level with the rest of the “Community” links. I’d prefer for the “Users” item to not even be visible for anonymous visitors. I tried to hide it with your JS code, assuming that it would only hide it when present under the main level of “Community”, but it also hides it when it’s present under “More” for logged in users:

const sidebarRemovals = [
  { sectionSelector: '.sidebar-section[data-section-name="community"] .sidebar-section-content',
    links: [
    	{ kebab: 'everything' },
    	{ kebab: 'about' },
    	{ kebab: 'faq' },
    	{ kebab: 'users' }
    ]
  },
  { sectionSelector: '.sidebar-section[data-section-name="community"] .sidebar-more-section-links-details',
    links: [
    	{ kebab: 'about' },
    	{ kebab: 'faq' }
    ]
  }
];

lol, i actually never tested it it in not logged in state! will add a feature to do what you want in a bit.

ok, first of all, there is SiteSetting.hide_user_profiles_from_public, which i am actually using. it has the additional effect of getting rid of the Users link from the sidebar for anon. does that fix your issue?

if not, under what situations (logged in, anon) do you and do you not want Users to show up? and in each case, do you want it in “…More” or not?

lastly, i am noticing some jank here because the “…More” thing is empty. it shouldn’t be there at all if it’s empty. do you want a feature added to the script to just remove “…More” and its contents? or maybe just remove “…More” if it’s already empty? maybe that would simply your sidebarRemovals as well.